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Introduction 



This report summarizes the activities and findings of the external evaluation of the Science: It’s 
Elementary (SIE) program during the period from July 2009 through June 2010. In the 
program’s fourth year, the evaluation collected data from a variety of sources using several types 
of instruments: 

Questionnaires 

• Administered a questionnaire to collect baseline data on the fourth cohort of participating 
teachers and schools; 

• Administered a post-professional development questionnaire to gather teachers’ opinions 
of the quality and impact of the professional development; 

• Administered an on-line module use survey to samples of participating teachers at the 
middle and end of the school year to examine how teachers are implementing the SIE- 
supported modules; 1 

• Administered an on-line survey to all participating grades 3-6 teachers at the end of the 
school year to capture how much instruction they provided on each of the topics on the 
student assessment and the extent to which that instruction was based on the SIE- 
supported modules; 

• Administered an on-line end-of-year questionnaire to a sample of teachers to gather 
feedback on the program and factors affecting their module implementation; 

• Administered an on-line questionnaire to principals of participating schools; 

Observations 

• Observed a sample of 25 professional development sessions to examine the quality of the 
training provided to teachers; 

• Conducted classroom observations of five teacher leaders and five non-teacher leaders to 
examine differences in their implementation of the SIE-supported modules; 

• Observed the 2010 ASSET Leadership Conference designed to prepare teacher leaders to 
provide professional development in future years of the program; 

Interviews 2 

• Interviewed a sample of 15 teachers participating in the SIE program; 



1 All of the on-line questionnaires were administered with the assistance of each participating school’s Support On 
Site person (SOS). SOSs were asked to disseminate the URL for the survey and follow-up with non-respondents. 

2 For each set of interviews, HRI drew a random initial sample from the targeted population and made repeated 
attempts to interview everyone in the sample. HRI randomly selected additional participants as needed to 
compensate for non-response. In an attempt to interview 20 teachers (5 from each cohort), 50 teachers were 
contacted repeatedly via email and telephone. Forty-seven principals were contacted in an attempt to reach the 
targeted goal of 20 (two principals responded to the interview request after replacements had already been contacted; 
thus the total number of principal interviews conducted was 22). HRI contacted 15 teacher leaders and 11 district 
supervisors in order to get 10 interviews from each group. 
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• Interviewed a sample of 22 principals whose schools are participating in the SIE 
program; 

• Interviewed a sample of 10 teacher leaders who had been trained by the program and had 
been serving as teacher leaders for at least two years; 

• Interviewed a sample of 10 district science supervisors/curriculum coordinators from 
Cohort 4 districts; 

Case Studies 

• Conducted classroom observations and interviews with a sample of 10 teachers to 
examine how they were using the SIE-supported modules and what factors affected that 
use; and 

Assessments 

• Administered student assessments in each grade 3-6 in all schools participating in the 
program. 

After a brief overview of the SIE program and the evaluation, this report describes the teachers, 
schools, and districts participating in SIE; the program’s work in developing teacher leaders; the 
nature and quality of the professional development provided by the program; and evidence of the 
program’s impact on teachers, their teaching, and their students. The report concludes with a 
summary of major achievements and HRI’s recommendations for the program. 
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Overview of Science: It’s Elementary 



The SIE program is managed by ASSET Inc. and overseen by the Pennsylvania Department of 
Education (PDE). SIE is an initiative aimed at improving elementary science instruction across 
the commonwealth of Pennsylvania. The program is focused on helping schools and districts 
implement an inquiry-based, hands-on science education program, with the ultimate goal of 
improving student learning. To accomplish its goals, SIE provides participating schools with 
module -based science instructional materials; teacher professional development around specific 
modules and inquiry -based science teaching more generally; and opportunities for strategic 
planning to help create supportive systems for science education reform. The program currently 
has five main components: 

1. Strategic Planning Institute: Based on the National Science Resource Center’s LASER 
model, these institutes are intended to help schools and districts understand, plan for, and 
successfully implement inquiry-based elementary science programs. Each school joining 
SIE sends a team comprised of administrators, teachers, and community partners to the 
Strategic Planning Institute to learn about science education reform and to develop a 
three-year strategic plan for their school. 

2. Teacher Professional Development: SIE offers three types of professional development. 
Teachers new to SIE attend a one-day Foundations course to learn about the program’s 
vision for science teaching. In addition, each teacher responsible for teaching an SIE- 
supported science module is expected to attend a one- or two-day Initial Module 
Training 3 focused on that module. This training provides teachers with an opportunity to 
familiarize themselves with the instructional materials; receive practical tips for using 
them in their classrooms; and learn teaching strategies, such as integrating science and 
literacy through the use of science notebooks. After teachers have implemented a module 
twice, they are offered a one-day Module Enrichment Training on that module to leam 
more about the targeted science content and how to implement the module more 
effectively. Teachers repeat this professional development cycle of Initial Module 
Training and Module Enrichment for each SIE-supported module they implement. 

3. Delivery of Classroom Science Materials: In addition to the professional development, 
the SIE program delivers science modules directly to schools for teachers to use in their 
classrooms. After instruction with the modules is completed, they are returned to ASSET 
where they are refurbished for later use. 

4. Leadership Conference: This three-day conference is intended to help develop the 
capacity of teachers selected by schools participating in SIE to become teacher leaders 
who will mentor and coach other teachers in their schools and districts. The program also 
expects that some of these teacher leaders will later be able to serve as facilitators of 
Initial Module Trainings. 



3 Most Initial Module Trainings are two days long; because some of the primary grades modules contain fewer 
lessons than the other modules, the Initial Module Trainings for these modules are one-day long. 
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5. Showcase of Student Learning: To help build community support for science education 
improvement, SIE encourages each participating school to host a Showcase of Student 
Learning. During this event, parents and other community stakeholders are engaged in 
science activities that highlight the hands-on, inquiry approach. These activities are 
typically led by students who have experienced this approach in their science instruction 
SIE provides materials and support to schools for hosting these events. 

Figure 1 shows the program’s theory of action, i.e., how program activities are intended to fit 
together to produce the desired short- and long-term outcomes. 
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Science: It’s Elementary Theory of Action 




Figure 1 



In addition, the SIE program requires each school to designate a Support On Site person (SOS) 
The SOS serves as the main conduit between the SIE program and the school, both for passing 
along important information from the program to the teachers and for helping teachers resolve 
any problems with the modules or professional development that may arise. The SOS is also 
responsible for registering teachers for SIE professional development and coordinating 
evaluation activities for the school. 



In Year Four, SIE added another cohort of schools to the program. A number of schools from the 
previous cohorts, primarily Cohort 1, stopped participating in the program. Anecdotal reports 
indicate that most of the withdrawing schools did so because they could not afford their share in 
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the cost-sharing required after their third year of participation. In total, over 2,500 teachers in 
138 schools participated in the SIE program this year. Table 1 shows the distribution of schools 
across the education regions in Pennsylvania. 



Table 1 



Distribution of Schools Participating in SIE 







Number of Schools 




Region 


Total 


Cohort 1 


Cohort 2 


Cohort 3 


Cohort 4 




r 1 


15 


0 


7 


1 


7 i 


East , 


f 2 


14 


2 


3 


4 


5 


1 


L 3 


8 


3 


3 


0 


2 




r 4 


17 


2 


4 


6 


5 


Central , 


5 


16 


7 


7 


1 


1 


1 


L 6 


22 


8 


10 


1 


3 i 


West 


r 7 


21 


6 


11 


1 


3 ! 


l 8 


25 


10 


8 


1 


6 


Total 


138 


38 


53 


15 


32 | 
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Overview of the SIE Evaluation 



The evaluation plan for SIE was developed by HRI in conjunction with key stakeholders from 
ASSET and PDE with the goal of examining the major elements of the program’s theory of 
action. The questions driving the evaluation focus on four main areas: (1) the development of 
an aligned system to support effective science education; (2) professional development for 
teachers; (3) the impacts of the program on teachers and their teaching; (4) and the impacts of the 
program on students. The key evaluation questions, by area, are: 

System Alignment 

1. How are school systems changing to support effective science instruction as a result 
of SIE? 

Professional Development 

2. What is the quality of the professional development provided to teachers? 

3. What is the quality and impact of the SIE leadership training? 

Impacts on Teachers and Their Teaching 

4. What is the impact of SIE on teachers’ preparedness to implement the modules? 

5. What is the impact of SIE on teachers’ implementation of the modules? 

Impacts on Students 

6. What is the impact of SIE on student achievement in science? 

This report provides data to help answer these questions based upon the program’s activities in 
Year Four. 
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The Context for Improving Science Instruction in SIE 

Schools 



Although helping schools and districts create fully-aligned systems is beyond the scope of SIE’s 
mandate, the school and district contexts in which the program is being implemented will likely 
factor into the success of the program. Data from teacher and principal questionnaires, as well as 
interviews with samples of teachers, principals and Cohort 4 district supervisors, shed light on 
how these contexts are impacting SIE’s efforts to improve elementary science education. 



Science Programs Prior to Joining SIE 

In the first three years, schools joining SIE have described their prior science programs as 
traditional and uninspiring, with a heavy reliance on lecture, textbooks, and worksheets. It was 
not uncommon for teachers to report that their schools had little, if any, set science program; 
decisions about what to teach, for how long, and in what ways were left up to individual 
teachers. As three teachers reported when describing science instruction in their school before 
SIE: 



Kids reading books, doing reports, not really doing a lot of hands on at all.. .what you 
would see more in a college setting. 

Just a book and a workbook; no experiments or anything. 

We did have a written curriculum in terms of concepts and skills, but the reality of it was 
that I would say the majority of classes handled it through reading and writing and not 
actual hands on. 

Principals gave similar descriptions of science instruction prior to SIE. As two reported in 
interviews: 

Before, they would read a passage from the textbook and answer questions. / think there 
were some labs in [the instructional materials], but I don ’t think the teachers ever did 
them. They didn ’t have the materials or maybe didn ’t know how. 

They were doing things out of a textbook, but not a lot of hands-on things. The teachers 
were not as comfortable with the material, so they may not have gone into as much detail 
as they do now about different topics. 

While schools in Cohorts 1-3 typically saw lots of room for improvement in their science 
programs, Cohort 4 teachers were generally more positive about the status of science instruction 
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in their schools before joining the SIE program. On the baseline questionnaire 4 , teachers were 
asked an open-ended question about the strengths of their schools’ science programs. The most 
common response, given by approximately 40 percent of the teachers answering the question, 
was the high quality of instructional materials available for teaching science. Often, these 
teachers referred to the availability and use of hands-on activities, though a sizeable portion 
indicated that their textbooks were of high quality. The second most commonly mentioned 
strength, given by about one-quarter of the respondents, was the flexibility teachers had for 
selecting topics and creating their own lessons. A similar number mentioned the availability of 
equipment and supplies for teaching science. Often teachers mentioned multiple strengths. 

These three comments illustrate the range of teachers’ responses: 

Textbook aligned with standards. Many science materials. Lots of FOSS kits. Staff. 
Hands on. 

Very flexible — no required texts, kits, or lesson; teachers had freedom to teach what they 
know, what they are interested in, and what resources are available in the 
school/commun i ty. 

We have a science resource room where we are able to gather materials (batteries, baking 
soda, potting medium) and equipment (microscopes, etc.). 

Although the use of hands-on activities was mentioned by a number of teachers, only about 20 
percent of the Cohort 4 teachers indicated having prior experience teaching science with a 
module like those supported by SIE. Of those with experience teaching with a module, nearly 
one-third had received no professional development on doing so; a similar proportion received 
training that lasted fewer than six hours. 

The baseline questionnaire also asked Cohort 4 teachers how their schools’ science programs 
could be improved. The three most common needs described by teachers, each mentioned by 
about one-third of respondents, were more supplies and materials for teaching science, 
incorporating more hands-on activities, and more instructional time for teaching science. About 
one-quarter of the respondents indicated that the science curriculum needed to be better aligned 
with state standards. In the words of four teachers: 

Science needs to be as important as reading and math and taught daily at all grade 
levels. Social studies and science are the first programs to be cut/shortened in an effort to 
spend more time on reading and math. 

Our science program can be improved by having more resources available for every 
grade level in our school. I also think that our schedule needs to allow more time in our 
day for teaching science. 



4 The baseline questionnaire was administered at the beginning of the Foundations courses. Because of the late 
funding of the program this year, the questionnaire was administered only in workshops that occurred after January 
1, 2010. HRI received responses from 487 participants. 
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Our school’s science program could be improved by having hands-on, easy -to-do science 
lessons and experiments. 

It would help if we had materials that matched the state ’s curriculum. Our textbook is 
outdated. I acquire my own materials, for the most part, in order to cover the content. 

Interviews with 10 district science supervisors asked how decisions about instructional time were 
made. Most indicated that these decisions were made at the district level; two said that it was 
left up to the discretion of individual teachers, and one indicated that building principals made 
the decision. Eight of the 10 reported that teachers in the upper elementary grades were expected 
to teach science every day of the school year, while two indicated that teachers were expected to 
alternate between science and social studies. There was more variation in expectations in the 
lower elementary grades, with half of the districts expecting teachers to include science 
instruction every day and half only a few times per week. 

The amount of time teachers are expected to devote to science on days when it is taught also 
varied widely. In the upper elementary grades, the expectation ranged from 30 to 50 minutes per 
day. In the lower elementary grades, at the lower end of the spectrum, some districts expected 
teachers to spend 30 minutes per day, three days per week (i.e., 90 minutes per week). At the 
higher end, some districts expected 40 minutes per day, five days per week (i.e., 200 minutes per 
week). 

However, most Cohort 4 teachers reported spending substantially less time on science than 
district supervisors’ expectations would suggest. Overall, most respondents to the baseline 
questionnaire indicated teaching science three days per week or fewer, though teachers in grades 
3-6 were more likely to report teaching science 4 or 5 days per week. (See Table 2.) In addition, 
the typical lesson was 30 minutes or shorter; lessons in grades 3-6 tended to be longer, with most 
being between 21 and 50 minutes in length. These numbers translate into an average of about 19 
minutes per day of science instruction overall; 12 minutes per day in grades K-2 and 27 minutes 
per day in grades 3-6. 
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Table 2 

Amount of Science Taught Prior to SIE 
by Cohort 4 Teachers, Overall and by Grade Range 





Percent of Teachers 


Overall 
(N = 487) f 


Grades K-2 Only 
(N = 269) 


Grades 3-6 Only 
(N = 210) 


Days per Week Taught 








0 


6 


7 


4 


1 


12 


19 


3 


2 


20 


26 


12 


3 


24 


24 


23 


4 


15 


12 


20 


5 


22 


11 


38 


Minutes per Lesson 








10 or fewer 


5 


9 


1 


11-20 


19 


29 


6 


21-30 


36 


45 


24 


31^10 


23 


14 


36 


41-50 


12 


3 


23 


51-60 


4 


0 


9 


61 or more 


1 


0 


1 



The overall number of respondents is greater than the number of grades K-2 and 3-6 teachers combined as 
eight respondents indicated teaching both grade ranges. 



Cohort 4 Schools’ Visions for their Science Programs 

In addition to characterizing the initial profile of their schools’ science programs prior to joining 
SIE, Cohort 4 principals and district science supervisors were asked in interviews to describe 
their vision for their science programs. All of the interviewees indicated that they would like to 
see their science programs incorporate more hands-on activities and become inquiry-based. This 
result is not surprising given that these schools chose to participate in the SIE program. In 
addition, all of the interviewed principals indicated a desire to incorporate more reading and 
writing into science. District science supervisors also mentioned making cross-curricular and 
real-world connections. Typical responses included: 

It would be very similar to the ASSET model where students are actually thinking; 
thinking about science, being posed a question or two, doing some research, getting a 
hands-on opportunity. (Principal) 

We want it to be hands-on, inquiry-based learning for all grade levels with our note- 
booking as a key component... we also want to infuse non-fiction reading and concepts 
that support the students on the PSSA. (Principal) 

My vision was that we link our science program to our literacy units and that we provide 
an inquiry-based opportunity for our kids to learn through problem solving and 
experiments. (District Supervisor) 
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District science supervisors were asked what the greatest needs of both teachers and principals in 
their district were in terms of accomplishing the vision. The most common need for teachers, 
mentioned by half of the interviewees, was professional development on how to teach science 
modules using inquiry techniques. Time to collaborate with other teachers and finding time to 
teach science were also mentioned. Often multiple needs were noted, as in the following 
example: 

I would say the biggest problem is training. They don 7 recognize the value of it. They 
were all educated in the same way we all did it. They don 7 know inquiry-based science, 
so if you don ’t know it you won ’t see the value of it and won ’t teach it that way. But 
outside of the training, it’s time. Give us the time and we’ll teach it. The training is the 
most important thing. Third, would be resources. Science education is not cheap, 
budgets are tight and we don ’t have resources. 

In regard to principals’ needs, interviewed district science supervisors most often mentioned: (1) 
time for observing and providing feedback to teachers; and (2) professional development for 
principals so they could learn about inquiry-based science instruction and give appropriate 
feedback to teachers. As one supervisor said when describing the needs of principals: 

They need to understand the philosophy as well. They need to understand what they ’re 
seeing, and when they do a walk-through they need to know what to look for, what’s being 
learned, and what questions to ask. They need to be involved and ask questions of the 
kids. . . They need to know the philosophy and the way to monitor that kind of learning. If 
the program is going to be successful, it depends on the building leader. 

Cohort 4 schools reported being at different stages in making their vision a reality before joining 
SIE. In interviews, most district science supervisors and a couple of principals noted that they 
had only partially reached their goal. In some cases, interviewees indicated that more progress 
ha been made in the upper elementary grades as these grades were more likely to include science 
instruction in the past. In each case though, SIE was seen as a major catalyst for continuing the 
progress, providing professional development and materials needed for schools to make changes. 
As a few interviewees reported: 

Thanks to the grant, we 're on our way. (District Supervisor) 

We were cdready moving in this direction prior to the grant, this [SIE] has just given us 
more resources. (Principal) 

It’s very laid out for the teachers, so simple for them to implement. They are busy and 
don ’t have to go find this and that, the kit is great to have it all ready for them. And the 
grant replenishes it for them, and that is a benefit. It’s all given to them. (District 
Supervisor) 
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Changes in SIE Schools’ Science Programs 



In Year Four of the program, Cohort 4 schools were just beginning the process of improving their 
science programs through participation in SIE, while Cohort 1, 2, and 3 schools were further 
along in the process. In addition to Cohort 4 district supervisors, samples of teachers and 
administrators, representing all four cohorts, were asked in interviews to comment on the 
changes that have occurred in their schools since joining SIE. Nearly all of the Cohort 4 district 
supervisors and principals from all cohorts who were interviewed indicated that science has 
become a higher priority in their schools since they joined SIE. However, many indicated that 
science was still taking a backseat to mathematics and reading/language arts. Representative 
comments include: 

The most profound change is the grade level at which we start. We have Kindergarten 
doing science now, and first and second grade are now doing science, and before, there 
was not any formed science in those grades. (Principal) 

Previously the priority on a scale of 1 to 10, it was a 1 — low. Currently, it’s an 8 or 9 
because we ’ve put focus on it because of the grant. (District Supervisor) 

Typically [science] is after math and reading, however we would like to integrate things 
so that we are more even and we are treating science just as importantly as math and 
reading. (Principal) 

It does not have the same priority as reading and math. At the state level the emphasis is 
on PSSA reading and math. We ’ve implemented a science test and that has ramped it up 
a bit, but it doesn 7 hold the same level as reading or math. [District Supervisor] 

One of the most telling indicators of the priority given to science is the amount of instructional 
time devoted to it. Based on questionnaire data collected at the end of the school year 5 , teachers 
in all four cohorts reported spending about 22 minutes per day on science instruction, though 
there is a large difference in the time devoted to science between teachers of grades 3-6 and 
teachers of grades K-2 (about 28 and 16 minutes per day, respectively). (See Table 3.) 



Table 3 



Average Number of Minutes per Day Devoted to Science Instruction, 
Overall and by Grade Range 





N 


Minimum 


Maximum 


Mean 


Standard 

Deviation 


Overall 


1753 


0.00 


226.67 


21.72 


19.18 


Grades K-2 


822 


0.00 


75.00 


16.32 


11.71 


Grades 3-6 


751 


0.00 


226.67 


27.65 


23.54 



5 This end-of-year questionnaire was completed by 1,580 of the 2,111 teachers sampled, a response rate of 75 
percent. 
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Sixty-eight percent of teachers indicated that they devoted more instructional time to science this 
year than prior to their school’s participation in SIE (73 percent of K-2 teachers, and 61 percent 
of 3-6 teachers). These teachers were asked how they made more time available for science. As 
can be seen in Table 4, integrating science with other subjects and reducing instructional time for 
social studies were each mentioned by over half of the teachers. Only a handful of teachers 
indicated reducing instructional time for reading/language arts or mathematics; 30 percent 
indicated cutting back on other subjects. 



Table 4 



Ways Teachers Adjusted Instruction 
to Increase Time for Science, Overall and by Grade Range 





Percent of Teachers Who Reported Increased Time for Science 




Overall 
(N = 1753) 


Grades K-2 
(N = 822) 


Grades 3-6 
(N = 751) 


Integrated science into other subjects 


54 


57 


48 


Spent less time on: 
Social studies 


53 


46 


64 


Reading/language arts 


9 


9 


8 


Mathematics 


8 


11 


3 


Other subjects 


30 


33 


25 



Responses add to more than 100 as teachers could select more than one option. 



In interviews, both teachers and principals elaborated on the increased time for science 
instruction: 

We are devoting quite a bit more time towards science than we had previously. Usually, 
we would allot 20 minutes a day, three days a week prior [to SIE ] and now it has 
increased quite a bit. We try to teach it 3-4 days and those time frames are longer only 
because the children need time to explore and get involved with the hands on teaching. 
(Teacher) 

Now, science is being taught in K-2, where it wasn 7 before. Grades 3-5, time has 
increased 5 to 10 minutes because the activities take longer to complete. Also there wasn ’t 
a deliberate teaching of science until we got into the program in grades K-2. (Principal) 

Devoting time to science has caused the teachers to spend less time on the other subjects. 
We ’re struggling with that now and how to work across disciplines to get that done. 
(Principal) 

There have been a number of other instructional changes in schools since they joined SIE. 
Principals report that the way science is taught has changed, and that instruction is now better 
aligned to state standards. As one principal said: 

Before [SIE] it was a hit or miss science curriculum. Some topics teachers did and some 
they ignored because of comfort level. Now with the PD, it has helped and we have 
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gotten proper training on the kits and focused some of our time as a science committee to 
look at gaps in our curriculum. 

Principals were asked to describe what a typical science lesson in their school looks like now that 
they are participating in SIE. Although descriptions varied somewhat, all included key 
components of the teaching approach that is central to SIE: active classrooms, hands-on, inquiry- 
based, and the use of science notebooks. Each was mentioned by several of the 22 interviewed 
principals. Typical responses were: 

It would be noisy, all over the place. ..the teacher moving about, children from life skills 
class would be involved in kit prep work, lots of documentation on the wcdls, they [the 
students] keep their science journal faithfully. 

Normally, they have an essential question posted and students are working in small 
groups on some type of experiment. Usually, you see a lot of data collection. 

Kids are seated in groups. They do a lot of cooperative learning. The teacher is a 
facilitator. She checks on each group and monitors their progress on whatever activity 
they do. Inclusion, as special education does this as well. Writing is a big piece with 
journals. The teacher is facilitating a thoughtful discussion with the class often. 

They start out with their notebooks. Some teachers have started using notebooks in other 
subjects, with the guided question. Depending on what it is, they ’re reading from one of 
the books that come with it or getting right into the experiment. And discovering the 
answer to the question without knowing beforehand. Then, they usually meet as a group 
and talk about the experiment and what they found out and write about it in their 
journals. 



Factors Affecting Science Program Improvement 

On a questionnaire, principals from each cohort were asked about factors that facilitate or inhibit 
their schools’ science programs. 6 As can be seen in Table 5, nearly all responding principals 
indicated that state science curriculum frameworks and access to professional development for 
teachers were facilitating factors. The availability and quality of resources for science 
instruction (presumably those being provided by SIE) were also frequently seen as helpful. 
Instructional time available for science and state and district testing in subjects other than science 
were the most frequently noted inhibitors. These results are very similar to those from previous 
years. 



6 The Principal Questionnaire was a web-based survey administered to the principals of all SIE schools. HR1 
contacted principals via email, following up with additional emails to non-responders. From the 138 schools in 
Cohorts 1^1, HRI received 113 responses to the survey, a response rate of 82 percent. 
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Table 5 



Principal Opinions of Factors that Affect Science 


Instruction 




Percent of Principals 
(N = 113) 


Facilitates 7 


Inhibits* 


State science curriculum frameworks 


92 


4 


Access to professional development for science 


90 


6 


Availability of resources for science instruction 


89 


6 


Quality of resources for science instruction 


89 


6 


Importance that the district places on science instruction 


87 


6 


District science curriculum frameworks 


85 


4 


State science testing policies and practices 


79 


11 


Consistency of science reform efforts with other district/school reforms 


72 


4 


District science testing policies and practices 


65 


4 


Instructional time available for science 


63 


29 


State testing policies and practices in subjects other than science 


54 


26 


District testing policies and practices in subjects other than science 


53 


15 


District grading policies and practices 


48 


4 


District policies for evaluating teachers 


45 


2 


District structures for recognizing and rewarding teachers 


25 


2 



Includes those indicating “Greatly Facilitates” and "Facilitates Somewhat” on a five-point scale of 1 “Greatly Inhibits to 5 
“Greatly Facilitates.” 



: Includes those indicating “Greatly Inhibits” and “Inhibits Somewhat” on a five-point scale of 1 “Greatly Inhibits to 5 
“Greatly Facilitates.” 



In interviews, principals and district science supervisors highlighted the role of the state in the 
development of their science programs through a variety of mechanisms: state standards; the 
Pennsylvania System of School Assessment (PSSA); and funding the SIE program. As several 
noted: 



I think the assessment for the state is driving part of it, with the fourth grade science 
exam... and just the resources to be able to do this. (Principal) 

The PSSA and the standards-cdigned system help at the state level because they 
[teachers] can narrow their focus and know what to teach and be able to sequence in a 
way that makes since. (Principal) 

Although the state investment in SIE was seen as a positive influence, concerns about continued 
funding, and the transition to a cost-sharing model, were often mentioned as inhibitors of 
progress. For example: 

The cut back on funding and having to cost share; the [school] board said they will pay 
cost share, but the board can change and I’m not sure how long we’ll be able to sustain 
it. (Principal) 

In order to continue participating, we are going to have to kick out a lot of money. I think 
if it were a small cost, we could do it but because it’s probably going to be large, we will 
not be able to participate. (Principal) 
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In addition, although adding science to the PSSA has increased the emphasis on science 
instruction, principals and district science supervisors indicated mathematics and 
reading/language arts are still the main priorities in their schools. One district supervisor 
described what might inhibit progress as follows: 

The PSSA; that’s flat out the answer. Everything boils clown to whether the kids can read 
and do mathematics by grade three... Previously, we cut science out of the program as we 
needed more time to teach reading... The fact that the science PSSAs don 7 count makes it 
so they don ’t care or try. There are no expectations at this level. 

Principals were also asked what additional services the SIE program could provide that would 
help them support their school’s science program. As can be seen in Table 6, ideas and tools for 
providing feedback to teachers, securing resources for science instruction, and enhancing teacher 
collaboration were each noted as potentially helpful by at least 50 percent of responding 
principals. Ideas and tools for assessing progress of the school’s science program and 
professional development for principals on how to integrate science with mathematics and 
literacy were also seen as valuable by nearly half of the principals. 



Table 6 



Principals Indicating that Potential Program Services Would be Very Valuable 





Percent of 
Principals 
(N = 113) 


Ideas and tools for observing and providing feedback to teachers about their science 




instruction 


54 


Ideas and tools for securing the resources needed for science instruction 


50 


Ideas and tools for creating opportunities for teacher collaboration 


50 


Ideas and tools for assessing progress of our school’s science program 


49 


Professional development for me to increase my own understanding of how to integrate 




science with mathematics and literacy 


48 


Ideas and tools for leveraging other district resources (e.g., professional development days) 




to support science instruction 


42 


Ideas and tools for increasing parental support/interest/involvement in our school’s science 




program 


40 


Ideas and tools for making science more of a priority in the school 


39 


Professional development for me to increase my own understanding of effective science 




teaching 


38 


Ideas and tools for increasing community support/interest/involvement in our school's 




science program 


36 


Professional development for me to increase my own science content knowledge 


20 
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The Quality and Impact of SIE Leadership Training 



In an effort to build capacity within the SIE program, both for providing professional 
development to a large number of teachers, and to bolster school-level support for the program 
and elementary science education improvement more broadly, SIE is developing cadres of 
teacher leaders. This section of the report describes the experiences of teacher leaders selected 
and trained in past years, including an examination of how their classroom implementation of the 
SIE-supported modules compares to non-teacher leaders. The following section addresses he 
quality of the three-day ASSET Leadership Conference (ALC) attended by new teacher leaders, 
using data from interviews with a sample of teacher leaders who had been trained prior to the 
2009-10 school year, classroom observations of a sample of five teacher leaders and five of their 
non-teacher leader peers, and HRI’s observations of the ALC held in May 2010. 



The Role of Teacher Leaders 

The primary role of an SIE teacher leader is to provide training and support for other teachers 
implementing the SIE science modules. Teacher leaders may be asked to do so informally, 
providing assistance to other teachers in their school and district, or more formally, helping with 
the facilitation of Initial Module Trainings for teachers around the state. To be successful in this 
role, teacher leaders need to know how to lead effective professional development, which 
requires them knowing how to purposefully implement the module themselves. 

Purposeful implementation of a module involves going beyond just doing the activities 
mechanically (i.e., following the steps in the teacher’s manual and having students complete the 
activities) to helping students make sense of the targeted ideas in each activity and how those 
ideas build toward bigger scientific principles. This type of implementation requires 
considerable knowledge and skill on the part of teachers. First, teachers need to know and feel 
comfortable with the mechanics of a module — “what’s in the box,” i.e., how to manage materials 
and students while using a module. 

Second, before they can effectively teach the science content to students, teachers must 
themselves understand the content. At a minimum, purposeful use of a module requires that 
teachers understand the ideas students are expected to learn, and how those ideas fit together in a 
coherent and cohesive conceptual framework. Teachers would also benefit from understanding 
the content beyond the student level, including how the ideas progress through the K-12 
sequence, so they can guide students along productive paths of inquiry. 

Third, in order to teach effectively, teachers need to have a clear understanding of how students 
learn science and an explicit vision of effective instruction. A great deal has been learned in the 
last few years about how people leam and the implications of research in the cognitive sciences 
for instruction. Key findings that have emerged from this research indicate that effective 
instruction provides motivation to learn, elicits students’ prior knowledge, engages students 
intellectually with important science ideas, emphasizes the use of data as evidence to 
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support/critique conclusions, and provides opportunity for students to make sense of the targeted 
ideas. 7 

For instruction to be purposeful and effective, teachers need to be able to integrate their 
knowledge of the science content and their familiarity with the nuts and bolts of the module. As 
a result, teachers would understand the “content storyline,” how each of the scientific ideas 
addressed by the module is developed through specific learning experiences. Ideally, they would 
be able to trace the development of a scientific idea through a module (e.g., students’ initial ideas 
about a particular idea are elicited in activity 1; the idea is then developed in activity 2; and 
students have an opportunity to practice and master the idea in activity 3). 

Similarly, for instruction to be purposeful and effective, teachers need to be able to integrate their 
knowledge of the science content with their understanding of how students learn science. This 
type of knowledge is commonly referred to as pedagogical content knowledge; it includes 
recognizing initial ideas students are likely to have that may get in the way of their learning the 
targeted concepts (often termed “misconceptions” or “naive ideas”), and specific strategies for 
helping students develop the correct ideas. For example, students often think that connecting a 
light bulb to one end of a battery will cause the bulb to light. The specific strategy for helping 
students move past this idea is to have them try many different configurations of connecting the 
bulb to the battery. Students record which do and do not work, and then analyze the results to 
find the commonalities in the configurations that work, and the commonalities in those that do 
not work. 

Finally, it is important to note that general pedagogical skills are necessary for effective teaching 
of any topic. The SIE program addresses two specific sets of general pedagogical skills: the 
integration of reading/language arts instruction with the teaching of science (e.g., notebooking 
strategies) and assessment strategies. Figure 2 shows a progression of how these elements build 
toward purposeful implementation of a module. 



7 National Research Council. (2000). How people learn: Brain, mind, experience, and school. J. D. Bransford, A. L. 
Brown, & R. R. Cocking (Eds.). Washington, DC: National Academy Press. 

National Research Council. (2005). How students learn: Science in the classroom . M. S. Donovan & J. D. 
Bransford, (Eds.) Washington, DC: National Academy Press. 
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Knowledge and Skills Needed for Purposeful Implementation of a Module 



Understanding the 
Mechanics of 
Implementing a 
Module 



Necessary for 



Understanding the 
> Content Storyline 
of a Module 



Understanding the 
Science Content 
Addressed by a 
Module 


Necessary for 




Necessary for 










Purposeful Use of 
a Module 


Necessary for 


Developing 


Necessary for 





Understanding 
How Students 
Learn 



Pedagogical 
t Content 
Knowledge for the 
Necessary for Topic 



Necessary for 



General Pedagogical/Classroom Management Skills 



Figure 2 



In an analogous fashion, teacher leaders need to know how to facilitate an Initial Module 
Training purposefully. Figure 3 shows what HRI believes teacher leaders need to know and be 
able to do to be successful in this role. The essential components needed for purposeful 
facilitation of an Initial Module Training are the same as those required for purposeful 
implementation of a module, with the focus being the content and activities of a workshop rather 
than a module. 
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Knowledge and Skills Needed for Purposeful Facilitation of PD for a Module 




Figure 3 



In order to help teacher leaders move from novice to purposeful facilitators of an Initial Module 
Training, the program has developed a Professional Development Facilitator Pathway. The first 
step in the pathway is attending an initial ALC, after which teacher leaders are asked to reflect on 
their growth and how they want to move forward. Next, the teacher leaders observe an In itial 
Module Training session; then, if they feel comfortable, they co-lead a session with an 
experienced facilitator. When ready, teacher leaders are approved by ASSET to lead professional 
development on their own. 



The Progression of Experienced Teacher Leaders 

Because the success of the SIE program depends, in part, on the success of the teacher leaders, it 
is important to understand the extent to which the teacher leaders are progressing as expected. 

To learn about their feelings of preparedness and experiences as teacher leaders, HRI interviewed 
a sample of 10 who have been teacher leaders for at least two years. All but one of these teacher 
leaders indicated teaching science at the elementary level prior to their school joining SIE, 
though only two had prior experience doing so with a module like those supported by SIE. In 



Horizon Research, Inc. 



21 



July 2010 



addition, 3 of the 10 had prior experience leading professional development for teachers, 
including 1 who had done so in science. 

Teacher Leaders’ Preparedness to Lead Initial Module Trainings 

The basic design of an Initial Module Training is to have participants experience the lessons in a 
module as a learner of both the materials and the science content. The sessions also include 
discussions of implementation issues and provide a number of suggestions for managing both 
students and the materials. Overall, teacher leaders indicated that they feel well prepared to lead 
these trainings, with all 10 interviewees attributing their preparedness to the training they 
received from ASSET, in some cases noting that their prior experiences with facilitation were 
helpful as well. As two said about the ASSET trainings: 

They were the backbone of my knowledge. You learn something new each time. 

Going to the [ASSET leadership] conference and learning how to train/facilitate, the 
questions, probing, and rules of collaboration have recdly set a solid foundation. 

The interviews probed teacher leaders about how well prepared they felt to help teachers in 
specific areas related to their module: learning the logistics of implementing the lessons; 
understanding the science content; and figuring out how to teach the module in the limited time 
available for science. Each interviewee indicated feeling well prepared to address teachers’ 
concerns about the logistics of implementing the module, with most of them attributing that 
confidence to their own experience teaching the module. A few of the teacher leaders mentioned 
that watching the ASSET staff model the workshop also contributed to their preparedness. As 
two shared: 

I have been teaching this module for three years now and have sort of a buy-in when I tell 
teachers, “7 know this works because I’ve used it in my classroom. ” 

Actucdly teaching it really helps because you have a recd-life experience to share with 
other teachers. 

There was more variation in regards to comfort addressing teachers’ content questions. Only one 
of the 10 interviewed teacher leaders reported feeling well prepared to address content questions. 
Several indicated that they were fairly confident, but relied on ASSET staff to address content 
questions. A few indicated that they felt comfortable with teaching the content to their students, 
but not addressing deeper content questions. One indicated not being comfortable with the 
science content in the module. The following quotes illustrate the range of responses: 

7 think 7 would be able to answer them [content questions] fairly well. On a scale of 10, 
about a 7 or 8. And I know where to guide them to go for further information. 

I feel confident with the part of the content that I would present to my first graders, the 
content base that they need to know. Some of the more advanced things, the “why” 
behind some of it, the more scientific part, I don i feel as confident because I only teach 
first grade. 
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I’m not really strong in science and would want to make sure my facts are all in order. 1 
would say something like, “I think I know, but let me double check. ” 



Given that the teacher leaders themselves indicate not feeling very confident with the science 
content, the program should consider ways to continue developing their understanding of the 
science related to their module, both to improve their own instruction and to be better able to 
help other teachers 

Interviewees were also asked what advice they would give a participant who is concerned about 
how to complete a module in the time allotted for science instruction. Responses varied greatly. 
Several indicated that they would simply advise teachers to “do the best you can.” Two indicated 
they would advise teachers to integrate science with reading instruction, in particular the 
notebooking aspects. One indicated that teachers should do as many of the lessons as they could, 
without skipping any; another indicated that teachers should pick and choose the lessons they felt 
most comfortable implementing. Two others indicated that they would advise teachers to pick 
the lessons most aligned with the state science standards. In the words of a few teacher leaders: 

7 would suggest they do the notebooking as their writing class that day. Or, the readings 
that some of the modules come with, using that in their reading class. 

My comment to that is always do as much as you can in the time you have... make sure 
you ’re covering the state standards because time is cdways an issue. 

I would say expose kids to certain parts and use your judgment on what parts to leave 
out. Instead of hitting things superficially, leave some things out. Do deep instead of 
covering things fast. 

These comments suggest that many, but not all, teacher leaders understand the program’s 
standpoint that teachers should cover as much of the module as they can sequentially with 
fidelity in the allotted time. SIE may want to address this issue explicitly in future ALCs and 
during Initial Module Training preparation meetings with veteran teacher leaders to ensure that 
participants receive a consistent message about the implementation of the modules. 

Teacher Leaders’ Experiences Preparing for and Leading Initial Module Trainings 

The interviews with teacher leaders included questions about how they prepare for, and their 
experiences in, leading an Initial Module Training. Eight of the 10 interviewees have facilitated 
at least one Initial Module Training. All of these teacher leaders indicated that they co-facilitated 
the workshop with an ASSET staff member, leading participants through some of the module 
lessons. They described their role as guiding participants through a lesson as if they were 
students, and providing implementation suggestions. For example: 

7 take the teachers through the different strategies for each lesson they are going to be 
presenting to their students within the module as though they were in their classroom; so 
they can see what they need to do and they need to have ready before hand; what they 
need to think about. 



Horizon Research, Inc. 



23 



July 2010 




When asked how it was decided what their role in the workshop would be, the teacher leaders 
indicated that the ASSET staff would typically ask what they felt comfortable doing. As three 
teacher leaders described: 



It was really about who felt comfortable doing what at what point. Everyday, we would 
plan before the next session. 

The first time, we picked lessons we felt comfortable with. 

The other person and I would talk it over or someone from ASSET would describe what 
lesson to do. 

When interviewees were asked what guidance they have received from ASSET regarding 
designing and implementing a workshop, all but one of the 10 interviewed teacher leaders 
who had co-led a workshop indicated that they learned primarily from observing and 
trying to emulate the ASSET staff. Only one teacher leader mentioned the Initial Module 
Training facilitators’ guide as providing guidance for their workshop. 

Overall, the interviewed teacher leaders valued the guidance and appreciated the supportiveness 
of the ASSET staff. In their own words: 

They are extremely helpful, honest, upfront, supportive, but you cdsofelt like you can ask 
them a question and they won ’t be like, “That was a dumb question. ” 

I recdly enjoy how they will sit down with me and give me tips and possible ways to 
improve, even after I present a session. 

Teacher Leaders’ Implementation of their SIE Module 

To examine the extent to which teacher leaders are purposeful in their implementation of their 
module, HRI observed five pairs of teachers implementing a lesson from their module. Each 
pair consisted of a teacher leader who had conducted at least one Initial Module Training in the 
module being observed and another teacher in the same school, ideally teaching the same 

o 

module. Information on the teachers and the observed lessons is shown in Table 7. 



8 Because funding for the program this year came through well into the school year, many of the teacher leaders had 
already completed their implementation of the module for which they provide training. Thus, HRI was not able to 
randomly sample teacher leaders for this study and the results may not be representative of all teacher leaders. 
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Table 7 

Observed Classroom Lessons 



School 


Teacher Leader 


Module 


Lesson 


A 


Yes 


Changes 


7 


Dissolving Race: Two Forms of Sugar 




No 


Changes 


4 


Mixing and Separating Solids 


B 


Yes 


Changes 


13 


Gas in a Bag 


No 


Changes 


10 


Separating Mixtures of Color 


C 


Yes 


Changes 


2 


Freezing and Melting 


No 


Changes 


2 


Freezing and Melting 


D 


Yes 


Electric Circuits 


8 


Making a Filament 


No 


Electric Circuits 


15 


Planning to Wire a House 


P 


Yes 


Mixtures & Solutions 


4.3 


Reaction in a Zip Bag 




No 


Electric Circuits 


13 


Constructing a Flashlight 



The classroom observations focused on the extent to which instruction provided opportunity for 
students to deepen their understanding of the intended science content. Interviews after the 
observations were used to gather information on the teachers’ experiences with the module and 
the strategies they use when implementing it. The framework HRI used for assessing student 
opportunity to learn is based on research on how people learn, as described earlier in this report. 
Intellectual engagement and sense-making are likely the most critical components of this 
framework, and are the ones teachers tend to struggle with the most. Following are the findings 
from a comparative analysis of lessons taught by teacher leaders and their non-teacher leader 
peers in each of these two areas, as well as the extent to which the teachers appeared to 
understand the content storyline of their module. 

Focusing Students on the Relevant Aspects of the Lesson Activities 

In order for students to learn new ideas, they must be intellectually engaged with those ideas, 
thinking and talking about the science content, and responsible for doing the intellectual work. 
Intellectual engagement involves focusing students on the relevant aspects of the lesson activity 
and understanding how data they are considering are related to targeted concepts. Students must 
know why they are doing the activity, or collecting the data that they are. In other words, 
instruction must be “minds-on,” not just “hands-on.” 

In the observed lessons, teacher leaders were more likely to intellectually engage their students 
with the targeted science ideas than were their colleagues. In the lessons led by teacher leaders, 
the students were focused on what they were doing in the lesson activity and why they were 
doing it (i.e., the activity was purposeful). All of the observed teacher leaders utilized focus 
questions in the lessons, which provided guidance to students about the purpose of activities. 

For example, in a Mixtures and Solutions module lesson, sixth graders were provided the focus 
question, “What might happen if you put calcium chloride, baking soda, and water in a zip lock 
bag?” This question let students know that they should be attending to any changes that occur as 
a result of mixing these components during the lesson. 

The teacher leaders’ questioning during activities also helped focus students on relevant aspects 
of the student activities. For example, a teacher leader asked, “What happened when you did 
that? Why do you think that happened?” In contrast, the observed lessons taught by non-teacher 
leaders were usually focused on the mechanics of doing the activity, with the teachers’ talk 
focused on procedure (e.g., “What should you do now?” and “Do the next step.”). For example, 
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in a Changes lesson led by a non-teacher leader, third graders were told they would be “doing the 
opposite of mixing colors” in a lesson where they added water to ink to separate it into its 
component colors. Instead of focusing students on the targeted idea that mixtures of liquids can 
be separated, the teacher focused students on the steps of the activity, saying things like, “Add 
two drops now.” and “Don’t let your markers touch.” 

Helping Students Make Sense of the Targeted Ideas 

In addition to engaging students with activities that are designed to develop their conceptual 
understanding, it is important that students be given opportunities to make sense of the targeted 
science concepts. Without this sense-making, students may simply do the activities without 
developing an understanding of the science involved, turning the lesson into “activity for 
activity’s sake.” The SIE professional development provides teachers with a number of sense- 
making strategies, such as the “line of learning” to help students make connections between the 
module activities and the content. 

In the observed lessons, teacher leaders were more likely to have a class discussion after an 
activity, They were also more likely to attempt one or more of the sense-making strategies 
introduced at the SIE trainings, such as having students revisit their initial answers to the focus 
question, write claims and evidence statements, or use the “line of learning. For example, after 
observing a chemical change where a gas was produced, third grade students had a brief 
discussion about what they observed then listened to the teacher (a teacher leader) read a science 
story. After the story, the students drew a “line of learning” in their notebooks and recorded 
something new they had learned. 

In contrast, the observed lessons led by non-teacher leaders tended to end with a recap of the 
activity experienced, but did not include structured time for sense-making of the targeted 
concept. For example, at the end of a second grade Changes lesson, the teacher asked students 
what they observed. A student replied that they had separated their mixture of salt and gravel. 
The teacher then asked students to clean up their stations and then sent them to recess. 

This differential focus on sense-making came through in the post-observation interviews as well. 
When asked what strategies presented at the IMTs had been most helpful in their implementation 
of the module, teacher leaders were more likely to mention strategies focused on uncovering 
student thinking and helping students reflect on the activity, while non-teacher leaders tended to 
mention time and/or materials management strategies. The following quotes illustrate the 
differences: 

Before 1 went through this program, often 1 would just have a kid answer and say okay 
good answer. Now, I am like, “Well, tell us a bit more ”. . .1 think kids need to hear kids 
answer things in their language sometimes versus the teacher telling them everything...! 
wait better, 1 am not so interested in giving my information, I need to know what they 
know. . . they are not always learning when we think they are, but if you let them talk you 
can kind of move away some of those misconceptions. (Teacher Leader) 
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What I found most helpful were the time management skills... I actually make up the 
science kit on their trays for them so that when I have a half hour to teach science at least 
that part is ready to go. (Non-Teacher Leader) 

Although teacher leaders were more likely to attempt sense-making, observers noted that they 
struggled to help students make the connections between what they did and the science ideas. 
Wrap-up discussions typically focused more on simply sharing observations from the activity 
and making sure students had the correct data. For example, in one teacher leader’s lesson the 
class discussion focused only on debriefing students’ results and did not progress to a thoughtful 
discussion about the meaning of their data. As the observer described 

Students in a lesson from the Changes module dissolved two forms of sugar (i.e., granules and a cube) to 
see which would dissolve faster. The idea to be learned from this activity was that a substance’s properties, 
such as size, determine, in part, if and how quickly it will dissolve. The post-activity discussion included: 

Teacher: Okay, I think we are ready to discuss our results. What did your group find? 

Student: The sugar cube won. 

Teacher: How about your group? 

Student: The grains. Oh, no, no; the cube. 

Teacher: Was it close or a long time? 

Student: It was close. 

Teacher: How about your group? 

Student: We couldn't tell, it looked like a tie. 

Teacher: If we did it again, would we get the same results? 

Some students say yes, some say no. The teacher went on to point out the group that had different people 
stirring each cup and noted that they might have different results if one person stirred both cups. The 
lesson ended without the class reaching the targeted idea and the teacher indicated that they would move on 
to the next lesson. 

Understanding the Content Storyline of the Module 

For both groups of observed teachers, the lessons focused more on completing the activity rather 
than on the concepts targeted by the lessons. In order for sense-making to occur, teachers must 
understand what science concepts are addressed by each lesson. One reason for the inadequate 
sense-making in the observed lessons may have been that the teachers did not have a clear 
understanding of which science concepts were targeted in the module lessons. When asked 
during interviews what they hoped students would learn from the observed lesson, most 
respondents, both teacher leaders and non-teacher leaders, described what students did in the 
lesson rather than a science concept or process skill. For example, one teacher leader indicated 
that the goal was for students to see a chemical reaction, rather than the idea that chemical 
reactions result in new substances. A non-teacher leader cited the purpose of the observed lesson 
was for students to, “Start constructing a flashlight with as little instruction as possible” instead 
of understanding the trade-offs of series and parallel circuits when designing a flashlight. 

The findings from these observations and interviews suggest that teachers participating in SIE 
would benefit from an increased emphasis on the science ideas targeted by the modules, as well 
as a clear understanding of what students should know and/or be able to do as a result of each 
lesson. Given the central role teacher leaders have in delivering professional development, the 
program may want to consider adding a check point to the Professional Development Facilitator 
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Pathway that asks teacher leaders to demonstrate an understanding of the content storyline of 
their module. 



Training a New Cadre of Teacher Leaders 

To continue building capacity to provide professional development to a large number of teachers, 
SIE recruited and begun the training of an additional cadre of teacher leaders this year. One 
major change to the process this year was the method for recruiting teacher leaders. In previous 
years, teacher leaders were often selected by administrators and sent to the leadership training 
without knowing what they were getting into. Consequently, the attrition rate for teacher leaders 
was fairly high. This year, aspiring teacher leaders completed an application process, which the 
program believes will result in greater buy-in and, thus, higher retention. 

Approximately 30 teachers selected to be part of this new cadre of teacher leaders attended a 
three-day ASSET Leadership Conference in May 2010. The program’s goals were to help 
teacher leaders: 

• Enhance their understanding of effective facilitation skills and reflect on areas of growth; 

• Develop an awareness of how five learning blocks are embedded in the modules; and 

• Begin a professional development pathway to follow throughout the 2010-11 school 
year. 

To accomplish these goals, the conference was structured around five key learning blocks: 

1. Leadership and Facilitation; 

2. Content; 

3. Inquiry; 

4. Literacy; and 

5. Assessment. 

With the exception of the content block which was dealt with only in module- specific breakouts, 
each block was addressed in a whole-group plenary session, and applied in module-specific 
breakout sessions. Following is an analysis of the extent to which the ALC met each of these 
goals based on HRI’s observations of the conference. 

Enhancing Teacher Leaders’ Understanding of Effective Facilitation Skills and Reflecting 
on Areas of Growth 

The plenary session on leadership and facilitation was the first of the conference and was 
comprised of two main parts. The first hour was devoted to the characteristics of effective 
leadership. During this portion, the teacher leaders were introduced to traits of effective leaders. 
They then watched a clip from a movie and were asked to identify how each of those traits was 
exemplified by the characters in the film. Finally, the teacher leaders were asked to select one 
trait to focus on during the conference, and were given opportunities throughout to reflect on 
how they have been developing that skill. They revisited this trait on the last day of the 
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conference, reflecting on how this leadership skill will help them both as a teacher and as an 
advocate for science education. 



The focus on effective leadership provided a foundation for the new teacher leaders’ role as 
leaders. Trying to identify the leadership skills shown in the film clip allowed the teacher 
leaders to reflect on the traits of an effective leader, pushing them to think beyond the viewpoint 
of a classroom teacher. Not unexpectedly, there was a great deal of variation in their 
interpretation of the film, with different teacher leaders interpreting the same action in the film as 
demonstrating different leadership qualities. Although each of the interpretations was valid, 
because multiple traits were present in each scene of the film, the distinctions among the traits 
may have been blurred. However, discussions about the traits later in the conference indicate 
that the teacher leaders were able to identify these traits correctly. 

At the beginning of the second part of this plenary session, the teacher leaders were briefly 
introduced to the norms of collaboration , 9 the elements of effective science instruction , 10 and 
effective questioning to promote understanding. After this introduction, they engaged in a nearly 
two-hour long “fishbowl” activity focused on facilitation of an Initial Module Training. The 
teacher leaders sat around the room and watched two ASSET staff members role-play the parts of 
facilitators preparing for and implementing an Initial Module Training. Other ASSET staff 
played the parts of participants in the Initial Module Training. The role play included a number 
of issues that commonly arise in professional development, such as participants who were off 
task or resistant to learning about the module. At various points of the role play, the ASSET staff 
would pause and ask the teacher leaders questions about what they observed, how the norms of 
collaboration or elements of effective instruction were being exemplified, or how facilitators 
handled tricky situations. 

The fishbowl activity was effective at highlighting the importance of session preparation, general 
facilitation moves, and working with a co-facilitator. It also provided the teacher leaders with a 
concrete example of the expectations for the look and feel of an Initial Module Training. 
However, teacher leaders were introduced to a lot of new information (i.e., the norms, the 
elements) and it appeared to be overwhelming. For example, during the “pauses” in the role 
play, when asked which norm was being used, teacher leaders would respond by naming one of 
the elements of effective instruction, or simply describe what the workshop “participants” had 
been doing. The program may want to consider ways to give new teacher leaders more time to 
process the different ideas, perhaps by asking them to read the relevant articles prior to attending 



9 The seven norms of collaboration are: ( 1) pausing; (2) paraphrasing; (3) probing for specificity; (4) putting ideas 
on the table; (5) paying attention to self and others; (6) presuming positive intentions; and (7) pursuing a balance 
between advocacy and inquiry. From: Garmston R. & Wellman, B. ( 1999) The adaptive school: A sourcebook for 
developing collaborative groups (pp. 37-49), Norwood, MA: Christopher-Gordon. 

10 The elements of effective science instruction drawn from research on how people learn are: motivation; eliciting 
students’ prior knowledge; intellectual engagement; use of evidence to critique claims; and sense-making. From: 
Banilower, E., Cohen, K., Pasley, J. & Weiss, I. (2008) Effective science instruction: What does research tell us? 
Portsmouth, NH: RMC Research Corporation, Center on Instruction. 
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the ALC, possibly noting ideas that they thought were particularly important and questions they 
had based on their reading. 

The new teacher leaders may have also benefited from a slightly more structured and detailed 
framing of the conference. Many of the teacher leaders’ comments and questions in this session 
were focused on the mechanics of preparing for and running an Initial Module Training. For 
example, questions included: “Do I need to bring my own module materials?” and “What time 
do I have to show up?” Although the program planned to (and did) address these issues later in 
the conference, the teacher leaders were not aware of this plan. The teacher leaders’ concerns 
about these issues may have interfered with their ability to focus on more advanced skills l ik e 
instituting the norms of collaboration or highlighting the elements of effective science 
instruction. 

Teacher leaders’ questions about their role were addressed on the last day of the conference via a 
Lead Teacher panel discussion. The panel of six veteran teacher leaders answered questions that 
had been submitted by the new teacher leaders during the leadership conference. Many of the 
questions dealt with logistical issues and the responses were helpful. For example: 

Question: How far can you defer from scripted facilitator’s guide? 

Response: Not at all. You can add your own style, but one should cover the material as written. 

Question: How much notification do we get before a session? 

Response: ASSET tries to givelO days notice. 

Question: When do you get a facilitator’s guide? 

Response: After your first observation. When doing a session, you get the guide and PowerPoints. 

Other questions focused more on the substance of facilitating a workshop. In contrast to the 
logistical questions, the responses to these were sometimes vague. For example: 

Question: How do we present module lessons? Do we do them as students or talk and discuss strategies? 
Response: You'll know when to walk participants through a lesson or when to just discuss it. You'll see 
for yourself when it is appropriate to do each. 

It is difficult to decide when and how to deal with the myriad of needs new teacher leaders have, 
and there typically are pros and cons of each approach. The program may want to consider 
incorporating a session on logistical concerns early in the ALC to allow teacher leaders to get 
these pressing needs addressed. Doing so may allow teacher leaders to better engage with the 
more challenging, substantive aspects of leadership and facilitation. 

Developing Teacher Leaders’ Awareness of How the Five Learning Blocks Are Embedded 
in the Modules 

While the leadership and facilitation block focused primarily on preparing teacher leaders to lead 
IMTs, the other blocks were intended to deepen their understanding of how to effectively teach 
their module. Improving their understanding of and ability to teach the module to students 
would also serve the teacher leaders well in leading IMTs. 
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The content block focused heavily on helping teacher leaders understand the content storyline of 
their module. Teacher leaders began by exploring their own prior knowledge of the content by 
responding to a formative assessment probe about the content in their module. Then the teacher 
leaders engaged in one of the hands-on activities from the module, followed by the introduction 
of the storyline of the module, and a discussion of where the activity fits conceptually in the 
module. The storyline was represented on concept maps the program has created (and which are 
used in the Module Enrichment professional development). At the end of the session, the group 
reviewed the PowerPoint slides about the science content in their assigned module. 

The inquiry block began with a plenary session that provided a definition of inquiry and an 
overview of the FERA learning cycle, 11 which is used in about half of the SIE-supported 
modules. During the Inquiry break-out sessions, teacher leaders examined a module lesson, 
identifying the FERA components, and then discussed the features of inquiry in their module. 

The literacy block focused on the connections between science and language arts. The plenary 
session had teacher leaders identify and create posters to visually represent what they saw as the 
connections. They shared their thoughts in a “gallery walk” during which the teacher leaders 
rotated around the room examining others’ posters. They also discussed the process of scientific 
argumentation and the importance of making thinking public for learning, which can be done 
through various written strategies. In the break-outs, teacher leaders examined ways to use 
science notebooks with their modules. The teacher leaders learned about how they could use the 
line of learning, claims and evidence, and power conclusion strategies with their module. 

The plenary for the assessment block included a discussion of both summative and formative 

1 2 

assessment, with an introduction to a number of key components of formative assessment. 
During the assessment breakout session, teacher leaders revisited the probe they had completed 
in the content block and discussed the use of probes to assess students’ thinking. They also 
discussed the importance of teachers stating the learning intentions of the module lessons and 
providing feedback to students as important components of assessment. 

Breaking into module -specific groups as part of each block was an effective strategy. It was a 
good way for the teacher leaders to see the materials (e.g., PowerPoint slides, notebook prompts, 
formative assessment probes) they will be using to facilitate an Initial Module Training. In 
addition, these break-out groups allowed the teacher leaders to work with the ASSET staff they 
will likely be paired with in leading an Initial Module Training, thus developing a connection 
with their future co-facilitators. 



1 1 The steps of FERA cycle are: Focusing on the content at hand through observations and questions; Exploring 
these ideas with hands-on experiences; Reflecting on what students have observed or measured to make meaning 
from their experiences; and Applying and extending their findings to new questions or problems. 

12 The key components of formative assessment that were considered are: (1) learning expectations and criteria for 
success; (2) questioning; (3) feedback; (4) self-assessment; and (5) peer-assessment. From, Black, R & Wiliam, D. 
“Inside the blackbox: Raising standards through classroom assessment,” Phi Delta Kappan, October 1998, pp. 139 — 
44, 146-48. 
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Overall, the learning block approach was sound. However, the inadequate science content 
knowledge of many of the teacher leaders may have interfered with their ability to engage with 
the blocks. For example, in one of the inquiry break-out sessions, the teacher leaders spent a lot 
of time attempting to complete and make sense of the selected module activity. Their struggle 
indicated a lack of understanding of the science in the module, and the session facilitator spent 
most of the time reviewing the content. The facilitator made the appropriate decision, as the 
teacher leaders could not have engaged with the discussion about inquiry in a meaningful way 
without understanding the content. However, by the time the teacher leaders understood the 
content ideas, there was little time left to discuss how the lesson exhibited the features of inquiry. 
Thus, the facilitator simply identified the components of FERA and process skills covered in the 
lesson for the teacher leaders. Given that many of the teacher leaders, past and present, have 
limited experience in science, the program may want to increase the time allotted for the content 
block in the ALC or consider other means for deepening the teacher leaders’ content knowledge. 

The different blocks provided the teacher leaders with a great deal of information and strategies 
they can use to supplement the teaching of their module. However, there was little guidance 
provided on when, in what ways, or how often these tools should be used in the IMTs. Given 
how much the IMTs need to accomplish, the teacher leaders would likely benefit from guidance 
in this area. The program may want to develop a storyline for each module’s Initial Module 
Training that specifies when and for what purpose to use each of the tools. 

Finally, as was the case with the leadership and facilitation block, the teacher leaders asked many 
questions about the logistics of their role throughout these blocks. The questions were good 
ones, and the ASSET staff did a nice job answering them, but these questions and answers 
diverted the focus from of the intended purposes of the blocks. 

Helping Teacher Leaders Begin a Professional Development Pathway to Follow 
Throughout the 2010-11 School Year 

On the first day of the ALC, the teacher leaders were introduced to the PD Facilitator Pathway. 
This session provided the teacher leaders with an overview of what training and support they 
would receive and the program’s expectations for them. Quite naturally, the session generated 
many questions from the teacher leaders. Most of the questions focused on the logistics of how 
the scheduling of the various aspects of the pathway would work (given their already busy lives 
as teachers). If the program decides to incorporate a session to address logistical concerns earlier 
in the conference, adding it to this session may be a natural fit. 



Horizon Research, Inc 



32 



July 2010 




The Quality of SIE Professional Development 



The primary aim of the SIE program is to enhance student learning of science in K-6 classrooms 
by providing high-quality instructional materials and professional development to help teachers 
implement those materials well. The consensus view among experts in the field is that 
professional development needs to be sustained over time, situated in teachers’ classroom 
practice, and focused on issues of curriculum and pedagogy related to the learning of specific 
content. Professional development should also model and then explicitly discuss effective 
instruction, and should involve multiple stakeholders within schools. 

The design of the SIE professional development program incorporates this wisdom of practice by 
providing teachers with a sequence of opportunities to move from mechanical (novice use) to 
purposeful implementation (expert use) of the instructional materials. The program includes 
three components. The Foundations course, intended for teachers new to the program, focuses 
on helping teachers develop a common vision of effective science instruction. Initial Module 
Training is intended to help teachers attain at least a mechanical use of the module, and to lay the 
groundwork for further professional development that will move them toward more purposeful 
use. Module Enrichment Training, offered to teachers after they have used a module for two 
years, is intended to deepen their understanding of the content storyline of the module and their 
pedagogical content knowledge around the topic of the module. 

During Year Four of SIE, Cohort 4 teachers attended the Foundations course and an Initial 
Module Training for their first module. Cohort 3 teachers attended an Initial Module Training 
for their second module, and Cohort 1 attended Initial Module Training for their third module. 
Cohorts 1 and 2 teachers, most of whom had taught an SIE-provided module for two years, had 
the opportunity to attend a Module Enrichment Training for that module. A post-professional 
development questionnaire was administered to teachers attending these workshops. 14 Table 8 
shows the number of teachers completing the questionnaire at each type of workshop. 



13 See for example: Elmore, R. F. (2002). Bridging the gap between standards and achievement: The imperative for 
professional development in education. Washington, DC: Albert Shanker Institute. 

Banilower, E. R., Boyd. S. E., Pasley, J. D., & Weiss, I. R. (2006). Lessons from a decade of mathematics and 
science reform: A capstone report for the local systemic change through teacher enhancement initiative. Chapel 
Hill, NC: Horizon Research, Inc. 

14 Due to a delay in funding, post-professional development questionnaires were administered only to SIE 
workshops held after January 1, 2010 and do not represent the entire population of SIE workshop attendees in Year 
Four. Because the post-professional development questionnaire was administered to a captive audience, the 
response rate should be close to 100 percent. 
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Table 8 



Responses to Post-Professional Development Questionnaire 





Number of Responding Teachers 


Initial Module Training 
Module Enrichment Training 


1,047 

57 



This section of the report describes the quality of each component of the SIE professional 
development program. Data sources include the baseline and post-professional development 
questionnaires, observations by HRI, and interviews with principals and teachers. 



Foundations Course 

The primary purpose of the Foundations course is to develop a common vision of effective 
science instruction for those teachers new to the SIE program. After an introduction to the SIE 
program, participants are asked to record their own ideas of what effective science instruction 
looks like. The participants post their ideas around the room, conduct a “gallery walk” (i.e., 
participants circulate around the room reading everyone’s postings), and then discuss what they 
read. Next, participants watch segments from the video “Minds of Their Own” in which MIT 
and Harvard graduates are asked questions about basic science concepts such as how to use a 
wire, bulb, and battery to create a complete circuit, revealing their lack of understanding of the 
basic concept of the flow of electric current despite their formal education. The video includes 
interviews with science education researchers that highlight the fact that there is a link missing 
between what teachers teach and what students learn. Participants also watch a video clip in 
which a high school physics teacher provides a clear and accurate explanation of what a 
complete circuit is, but at the end of instruction his brightest student still does not understand 
how current flows in a simple circuit. 

After watching and discussing the videos, participants are introduced to the elements of effective 
science instruction: motivation; elicitation; intellectual engagement; claims and evidence; and 
sense making. For the remainder of the session, participants engage as learners in a science 
activity on electromagnets that models these elements. After the activity, the facilitators 
highlight when and where they modeled each element of effective instruction. The session ends 
with participants watching a final video clip of the high school physics teacher’s class and how 
his instruction had changed because of what he discovered about how students leam. 

There were a number of strengths in this year’s Foundations courses. In particular, the group 
discussions of participants’ own ideas of effective science instruction and video examples were 
effective at highlighting the SIE program’s vision. For example, participants were observed in 
one session having a thoughtful discussion about how they entered the training believing that the 
modules were primarily about having students do hands-on activities. They noted that the 
discussion about effective science instruction and the video clips changed their opinion; they 
indicated that because of the Foundations course, they understand that the program was 
promoting a higher level of student intellectual engagement in learning science. As one 
participant stated: 
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Given the time to now see what this is really about, 1 now understand you have a different 
view. It’s not just, “Here are all the materials now go experiment. ” It needs to be more 
in-depth. 

However, there was considerable variation both within and across sessions in the facilitators’ 
skill at leading this training. In many of the observed sessions, the participants struggled with 
the electromagnets activity due to a lack of initial guidance for the task. In sessions where the 
activity worked well, the facilitators took time at the beginning of the activity to review what a 
complete circuit is and how electric current flows in an electromagnet. 

In other sessions, because participants did not understand how an electromagnet worked, they 
struggled with how to assemble the electromagnet and understanding why the variables being 
investigated would make any difference in the strength of the electromagnet. In these sessions, 
the participants tended to fall behind, struggling with the activity, and the facilitators faced a 
difficult choice of pressing ahead to stay on schedule, or slowing down but having to cut 
something from later in the session. Typically, the facilitators chose to move forward, which 
resulted in some participants not having the opportunity to get beyond the mechanics of the 
activity and consider how the activity could illustrate the elements of effective instruction. 

Although it is a good idea for participants to engage with an idea where they do not necessarily 
know the correct outcome, the program may want to include a brief explanation of 
electromagnets as a standard part of this activity. Alternatively, the program may want to 
consider selecting an activity for which teachers are more likely to have the prerequisite 
knowledge in order to illustrate the elements. 



Variation in the skill of the facilitators was also observed in the portion of the session that 
discussed how the electromagnets activity illustrated the elements of effective instruction. In 
some sessions, the facilitators were able to guide participants to correctly identify the elements 
For example, in the discussion about the elements in one session, the facilitator first reminded 
the participants what each element was then asked them for examples from the activity using 
questions to check for understanding: 



Facilitator: 

Participant 1: 
Participant 2: 
Facilitator: 

Participant 2: 
Facilitator: 
Participant 3: 
Facilitator: 



Now you need to keep the kids intellectually engaged. That is, keep them focused. Where 
did we use intellectual engagement? 

When we did the lesson. 

The hands-on. That’s engaging. 

Okay. The hands-on; the doing the activity. Can you say more? Did just doing the hands- 
on work keep you focused? 

We were focused on doing the work. 

But you need to be focused on the work and on? 

The science? 

Yes, the science. Kids need to not just do the activity but be doing the science. The 
questions we gave you helped keep you focused on the science when you were doing the 
hands-on. 



However, in other sessions, the elements were sometimes misrepresented. For example, in one 
session the facilitator told participants, “Building the electromagnet was the engagement.” As 
the purpose of the activity was to determine the factors that influence the strength of an 
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electromagnet, simply building an electromagnet was not an example of “intellectual 
engagement.” Rather, the controlled testing of the different factors, and the careful recording 
and organization of data were the key parts of the activity that allow the learner to engage with 
the targeted idea. 

To help ensure greater consistency in the quality of these sessions, it will be important for the 
program to consider ways to help all facilitators of the Foundations course develop a clear 
understanding of the elements themselves and of how the selected activity exemplifies those 
elements. 



Initial Module Trainings 

The primary purpose of the IMTs is to prepare teachers to use an SIE-supported module for the 
first time, including familiarizing them with the activities, the targeted science content, and 
strategies for engaging students with the content. The workshops often use the “learner 
hat/teacher hat” approach in which participants engage with an activity from the module as 
learners and then discuss the activity as teachers. The sessions also provide participating 
teachers with opportunities to learn a variety of instructional strategies that can be used when 
implementing the module; in particular, the workshops emphasize the use science notebooks as a 
way to integrate literacy skills into science teaching. Data on the quality and impact of the IMTs 
come from questionnaires completed by teachers attending these trainings, interviews with 
samples of teachers and principals, and HRI’s observations of a sample of workshops. 

Teacher Expectations 

Knowing what participants’ needs and expectations are for a professional development 
experience can provide valuable insight for understanding their perceptions of the quality of that 
experience. An open-ended item on the baseline questionnaire asked teachers new to SIE what 
they hoped to gain from participating in SIE professional development. The most common 
response, given by approximately one-third of the respondents, was that they wanted to learn 
hands-on activities and experiments to use in their science teaching. Over one-quarter of the 
questionnaire respondents indicated that they wanted to learn how to make science more 
enjoyable and engaging for their students. Typical responses included: 

Knowledge of easier science experiments that are effective. 

Methods/techniques to get kids and staff excited about teaching a “hands on ” science 
program. 

I’m hoping to walk away with new and fun ways to teach science, as well as, resource 
materials to use. 

Teacher and Principal Perceptions of the Quality of the Initial Module Trainings 

Teacher feedback on the post-professional development questionnaire administered at the end of 
each training was overwhelmingly positive. One area in which the workshops continue to 
receive high ratings is the quality of the facilitation. Nearly all responding teachers thought the 
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facilitators provided useful suggestions for successfully implementing the modules, encouraged 
active participation, and helped address questions about the science content in the modules. In 
addition, the vast majority of teachers indicated that the Initial Module Training sessions 
increased their familiarity with the activities in the modules and prepared them to use the 
teaching strategies promoted by SIE. Teachers also indicated that the workshops had increased 
their understanding of the science content in the modules. (See Table 9.) 



Table 9 

Teacher Opinions of the Quality of SIE Initial Module Training 





Percent of Teachers 
Agreeing 
(N = 1,045) 


This workshop familiarized me with the activities and materials in the module. 


99 


The facilitators encouraged active participation and investigation by all participants. 


97 


The facilitators shared tips and suggestions for successfully implementing the module. 


97 


The goals of this workshop were clear. 


97 


The various components of this workshop were useful in meeting its goals. 


97 


This workshop increased my understanding of the science content in the module. 


97 


There were opportunities for participants to express their views and collaborate with peers. 


96 


This workshop reflected careful planning and organization. 


96 


Question about the science content in the module were adequately addressed by the 




facilitators. 


95 


The facilitators modeled effective teaching strategies. 


95 


The facilitators were well prepared. 


95 


This workshop increased my confidence in my ability to teach using the SIE module. 


94 


This workshop prepared me to use the teaching strategies promoted by SIE. 


94 


This workshop was relevant to my classroom instruction. 


94 


I would recommend this professional development to a colleague. 


93 


The facilitators explicitly discussed how, when and why to use different teaching strategies. 


93 


This workshop was worth the time that I invested. 


93 


Adequate time, structure, and guidance were provided for participants to reflect individually 




on the substance of this workshop. 


89 


Adequate, time, structure, and guidance were provided for participants to discuss the SIE- 




supported modules and pedagogical strategies with each other. 


89 



Includes those teachers agreeing or strongly agreeing with each statement. 



In addition, approximately three-fourths of respondents indicated that the pace of the workshop 
was appropriate. (See Table 10.) Fourteen percent of participating teachers thought that the pace 
of the IMTs was too slow; 12 percent indicated the pace was too fast. 



Table 10 



Teacher Opinions of the Pace of SIE Initial Module Trainings 





Percent of Teachers 
(N = 1,027) 


Too slow 


14 


Appropriate 


74 


Too fast 


12 
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When asked on the post-professional development questionnaire which aspects of the IMTs were 
most effective in preparing them to implement the module, over 60 percent of respondents to this 
open-ended item highlighted going through the module activities as if they were students. 
Receiving useful suggestions for implementation from the facilitators was mentioned by about 
one-third of the respondents. These sentiments were echoed in interviews with teachers. In the 
words of three: 

7 enjoyed being able to go through step-by-step the different lessons in the module. It 
gave me a clearer understanding of what my kids are expected to do. 

I really like the way they walk us through the lessons. ..so we can actucdly see it in action. 
I recdly appreciate that because when I walk into my classroom I feel like I have already 
seen it, lam comfortable with it, and 1 can execute it... I like the fact that they had 
teachers coming to do the training because they have actually done it in the classroom 
and they know what works and what does not work, so it was a lot more realistic point of 
view. 

They [the facilitators] were very knowledgeable about the content of the module and 
offered tips for the best way to implement it in the classroom. 

When asked how the IMTs could be more effective, teachers’ most common response was that no 
changes were needed. When teachers did have suggestions for improvement, the most common 
were to lengthen the workshop to allow a more in-depth coverage of the module and to allow 
more time for teachers to collaborate with their peers. As two wrote: 

A little more modeling of course questions or at least discussion after [the activity]. ..A 
little more wait time to write in notebook. The second day seemed very rushed. 

More discussion between other teachers/districts on how they would adapt with how 
much time they have to teach science. 

Another indicator of the quality of the professional development is what concerns participants 
had about implementing the module in their classroom after the training. Very few participants 
had concerns about their ability to teach the module. The most common concern, mentioned by 
nearly half of the participants responding to this open-ended question, centered on time; either 
finding instructional time for science or finding time to prepare all the materials needed to use 
the module. Managing the materials that come with the module was mentioned by about 1 in 5 
respondents. As two teachers wrote: 

There is not enough space in my classroom for cdl this material! We are also on a very 
tight schedule and there will not be enough time to complete all activities. At this point, 
we are lucky to have one day a week to fit in science — not V 2 hour each day as suggested. 

Being able to fit cdl of the lessons into the time that will be left over after teaching other 
core subjects. 
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Observers’ Perceptions of the Quality of the Initial Module Trainings 

Evaluation staff observed at least a portion of each of the 16 Initial Module Training sessions 
between January and February 2010. HRI attempted to observe IMTs for as many different 
modules as possible, at the same time maximizing the number of workshops observed in each 
trip. Table 11 summarizes the number of sessions observed for each module. 



Table 11 



Initial Module Trainings Observed 





Number of Sessions 


Changes 


1 


Ecosystems 


1 


Electric Circuits 


3 


Levers and Pulleys 


2 


Magnets and Motors 


1 


Motion and Design 


1 


Organisms 


1 


Plant Growth and Development 


1 


Rocks and Minerals 


1 


Variables 


1 


Water 


1 


Weather 


1 


Wood and Paper 


1 



HRI’s observations focused on the extent to which the IMTs: (1) prepared teachers to implement 
the modules at the mechanical level; (2) deepened their understanding of the content in the 
module; (3) made teachers aware of the science content students should learn in each module 
lesson (i.e., the content storyline); and (4) deepened their understanding of effective science 
instruction. 

As described previously, the main professional development strategy used in many of the IMTs 
was the learner hat/teacher hat approach. In this approach, participants first engage in a module 
activity from the student perspective and then discuss the design and facilitation of the activity 
from the teacher perspective. This approach is particularly well suited for the Initial Module 
Training sessions because of their multiple goals of deepening teachers’ content knowledge and 
preparing teachers to implement the modules. The use of this method creates a structure to, and 
expectations for, the professional development that increases the likelihood that all goals will be 
addressed adequately, and that the science content will not get lost. 

As with any approach, this one involves a number of trade-offs. One is that it requires 
facilitators to go slower through individual module activities, spending time both modeling 
lessons (learner hat) and debriefing lessons (teacher hat). Consequently, facilitators may not be 
able to cover all lessons in a module. One way to address this issue is to use this approach only 
for selected lessons, those that cover science ideas that are particularly challenging and/or 
important, and those that are the most difficult to implement from a logistics standpoint. For 
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other lessons, facilitators would go faster, perhaps simply describing the lesson briefly or asking 
participants to review it on their own (e.g., in the evening after the first day). 

In some of the observed IMTs, the facilitators used the learner hat/teacher hat approach quite 
skillfully. In these sessions, the facilitators were very deliberate and clear about when teachers 
should be in which hat. The facilitators also made sure to devote enough time to each phase for 
the participating teachers to engage meaningfully in the substance of that mode. In addition to 
“cueing” the participants as to when they were in learner mode or teacher mode, the most 
successful sessions addressed each of the targeted goals through the use of the learner hat/teacher 
hat strategy. 



One example of the effective use of the learner hat/teacher hat strategy was observed in an 
Ecosystems session. In this session, the facilitator led the participants through the activity and 
made sure they made sense of the science content themselves before stepping into teacher mode 
to talk about implementation issues. As the observer described: 



In the learner hat mode, participants completed the student activities of reading from the science text that 
accompanies the module about gas exchange between plants and animals. They also created a web to 
visually represent the relationships among the biotic and abiotic things in an aquarium they had set up 
based on evidence from the reading and observations. The group then looked at the focus question for this 
lesson, “How do snails, fish, and plants together affect their environment?” The facilitator introduced the 
sense-making strategy of writing a “claims and evidence” statement, by providing the sentence starters, “I 
claim that. . .1 know this to be true because. . .” Participants shared their own claims and evidence 
statements, and the facilitator discussed what was good about their statements and what was missing. Part 
of the discussion included: 



Participant: 

Facilitator: 

Participant: 

Facilitator: 

Participant: 

Facilitator: 

Participant: 



Plants are important to fish because fish are alive. 

Good claim, but you need to cite some type of evidence. How do you know that 
plants are important to fish? Why are they important to fish? 

They give off oxygen in photosynthesis. 

And what does oxygen do for the fish? 

They breathe it. 

Good, so do you want to try your claim and evidence again with this information? 
I claim that plants are important to fish because plants give fish oxygen to breathe. 



Several other participants made claims and evidence statements, with the facilitator pushing them to cite 
their evidence appropriately. The facilitator then switched the conversation to teacher hat mode, and 
stressed that, “You shouldn’t let them [students] say, ‘because I know it.’ Students have misconceptions 
and you can get to them if you ask them their reasons why.” 



The facilitator also provided information on why the “claims and evidence” strategy is useful for sense- 
making and specifically how to use it in this lesson. 



Facilitator: Leave these webs up and you’re seeing if they can interpret these webs. . .A lot of 

times kids will put up arrows [to show relationships on the web] and they don't know 
what’s going on, but the writing is getting at what they really know and understand. 
The writing is really crucial for learning. 



The facilitator also suggested that the teachers model making claims and evidence statements first and do 
them as a whole class, then transition to having students do them in pairs before having them do it 
individually. 
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In another case, the facilitator of a Changes workshop made use of the teacher hat mode to make 
sure the participants understood what students should leam from the lessons they just 
experienced: 



Facilitator: 



Participant 1: 
Participant 2: 
Facilitator: 



Participant 3: 
Facilitator: 
Participant 4: 
Facilitator: 

Participant 5: 
Facilitator: 



Think about the three lessons that we’ve already done. The change cards, then the freezing 
of the ice cube. We saw that the water from the Petri dish [disappeared]. What was the 
content in this investigation? 

Gases 

Evaporation 

Right, so we wanted to show them that water can change from a liquid to a gas — the water 
cycle. So what exactly would you want your students to understand after these three 
lessons? 

The three properties of matter: solid, liquid, gas. 

So the three states of matter: solid, liquid, and gas, and the water cycle. 

Flow do we show them the back again? The gas to a liquid? 

We showed them that when we put the cup over the warm water and saw the condensation. 
What else? 

Heat. 

When you apply heat or take heat away is what changes the properties of the states of 
matter. So that’s what you want them to know. . .There’s a lot of learning in these lessons. 



In a workshop on Electric Circuits , the facilitators used specific reflection questions during the 
teacher hat phase to help participants think about purposeful implementation of the module: 

After having participants complete an activity as students, the facilitators asked the participants to consider 
the following questions about the teaching of the lesson: 

o How will your students reflect? 
o What will your students record in their notebooks? 
o How will you teach this lesson? 

o How does this lesson connect to previous lessons and the big ideas? 
o What are key vocabulary words? 

After participants had time to think about the questions individually, the facilitators led a short discussion 
about the participants' thoughts. The following is an excerpt from the discussion: 

Facilitator: How does this lesson connect to previous lessons? 

Participant: You have to know how to do this to do this, [gestures on paper] 

Facilitator: She’s saying previously in their drawings when they were talking about the complete 

pathway, they were using the bulb, the cell and the wires. . .they are using the same 
components but now with the secret language. 

These questions and the discussion helped the participants reflect on how they were going to implement the 
lesson and provide structure to help students leam the intended concepts. 



However, there was a great deal of variation across the observed sessions in implementing the 
learner hat/teacher hat approach. In some sessions, the facilitators were not careful about 
transitioning between the two hats, quickly moving back and forth between the two within a 
single lesson, which made it difficult for participants to fully engage as either learners or 
teachers. 
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In other sessions, the learner hat phase was implemented effectively, allowing participants to 
experience the module lessons and engage with the content as students would, but the teacher hat 
phase was not used to its full potential. In some of these cases, the facilitators used the teacher 
hat phase primarily for discussing logistical issues, spending little, if any, time on reinforcing the 
targeted science ideas or how the lesson develops those ideas. In other cases, the facilitators 
shared a strategy for helping students make sense of the targeted ideas, but neither modeled or 
discussed an appropriate use of that strategy. For example, in a Motion and Design session, 
participants put on their teacher hats to learn about a specific strategy but were not provided 
adequate guidance on the proper use of the strategy: 

The participants in this session completed a lesson from the module where they attached a falling weight 
system to a toy car and adjusted the force pulling the car by adding metal washers to the falling weight. 

The facilitator introduced the “claims and evidence” strategy at this point by describing it as “a sense- 
making and reflection on what we’ve done that’s not in your manual.” A handout was given to participants 
that described the strategy and included the sentence starters, “I claim that... I know this to be true 
because. . .” The facilitator then instructed participants to, “Take a look at your data collected and see if you 
can make a claim to explain your results.” After a couple of minutes for individual writing, a few 
participants shared their statements: 

Participant 1 : I claim that the more force, the vehicle will move faster and faster. I know this to 

be true because my data table showed it going faster. 

Participant 2: I claim that more washers will increase the speed of the car. I know this to be true 

because when 1 added more washers the speed increased. 

After these statements were shared, the facilitator stated, “I claim that increasing the force will increase the 
speed because more washers made the car move faster.” The facilitator then moved on to the next part of 
the workshop, with no discussion of these different claims and evidence statements (all of which contained 
the misconception that force is related to speed as opposed to the correct idea that force is related to a 
change in speed). Nor was there any discussion of what the teachers should do with students’ claims and 
evidence statements. 

The purpose and proper use of various instructional strategies, such as the “claims and evidence” 
strategy described, may seem obvious to the facilitators, but it is unlikely that participants will 
intuit the proper use without further assistance. Providing teachers with concrete examples of 
what students might say or do when using the strategy and what the desired outcome is (e.g., not 
only what a correct “claims and evidence” statement is but also why it is correct) would increase 
the likelihood that teachers will be able to implement these strategies effectively in their own 
classrooms. 



Because of time constraints, little time was left after the learner hat phase. Moreover, what little 
time there was for considering the lessons from the teacher perspective was primarily devoted to 
logistical issues. Consequently, there were few opportunities to reinforce participants’ 
understanding of the elements of effective science instruction. Several of the IMTs modeled the 
elements during the “learner hat” phase, and occasionally a facilitator would explicitly discuss 
them. However, the discussion tended to be shallow. For example: 



Facilitator: 

Participant: 

Facilitator: 

Participant: 



Where did we have motivation? 

That was the focus question. 

Where did we see intellectual engagement? 
That was the activity. 
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While not incorrect, the responses to the questions are vague and did not allow the facilitator to 
know if the participants understood what aspect of the activity was key for engaging students 
with the targeted science idea. 

In a few cases, the facilitators did not consistently model effective instruction during the learner 
hat phase. For example, in an Electric Circuits session, the participants were asked to respond to 
the question, “How are the parts of a light bulb important for making it light?” after completing 
an investigation in which they examined the inside of a light bulb. Two participants shared their 
responses to the question, and then the facilitators moved on to the next lesson without checking 
whether all participants had a common understanding. 

In other cases, participants were not given the opportunity to make sense of the science in an 
activity at all. Sometimes the facilitators would tell the participants that it was important that 
they help students reflect on the science, saying, “This would be a good place to answer your 
focus question,” or “a claims and evidence could be done here.” These comments indicated they 
were aware of the importance of sense-making, but they typically did not take the time to make 
sure that the teachers understood the science. In another example, the facilitator’s questioning 
focused on having the participants report what they had observed but without asking them to 
interpret their observations or draw conclusions. 

SIE appears to be on the right track with the learner hat/teacher hat approach. However, it needs 
to continue to work with all current and potential facilitators to make sure they have the 
knowledge and skills to implement it effectively. 



Module Enrichment Trainings 

Cohort 1 and 2 teachers, after having taught a module at least two times, had the opportunity to 
attend a one-day Module Enrichment Training for that module. These workshops are intended to 
move teachers further along the continuum from mechanical to purposeful use of the modules. 
The focus is on deepening teachers’ understanding of the content storyline of a module, as well 
as their pedagogical content knowledge (e.g., what misconceptions student have about the 
content in the module and how teachers might help them change their ideas). The program’s 
stated goals for these workshops are to: 

1. Uncover and address teachers’ concerns about the module; 

2. Examine a module specific storyline that connects instructional tasks, learning goals, 
and big ideas; 

3. Establish a clear understanding of what we expect students to learn in that lesson, 
how we know what they learned, and what we can do if learning has not occurred; 

4. Discuss “science misconceptions” and identify any that may affect the learning goals 
of the module; and 

5. Link storyline content to the PSS A. 
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The enrichment sessions begin with teachers sharing and discussing concerns about the modules 
based on their experiences using them. After this discussion, the participants engage in an 
activity to learn how to read a “concept map” of their module. The maps, created by the project, 
included instructional tasks (things students do in the module) linked to learning goals (ideas 
students will leam) linked to the big ideas of the module. After the introduction to the map, 
participants spent the majority of the session in a lesson-by-lesson review of the module, 
mapping lessons to the learning goals. There are also discussions about science misconceptions 
students may hold and how teaching the module assists students in answering PSSA questions. 

This section of the report describes the quality of the SIE Module Enrichment professional 
development using data from teacher questionnaires, teacher interviews, and observations of a 
sample of Module Enrichment sessions. 

Teachers’ Perceptions of the Quality of the Module Enrichment Trainings 

A majority of participating teachers had positive opinions about the Module Enrichment 
Trainings. On a post-professional development questionnaire administered at the end of the 
training, participants indicated that they considered the facilitators to be of high quality. For 
example, nearly all participants thought that facilitators were well prepared, encouraged active 
participation, shared tips for implementing the module, and addressed questions about the 
content in the modules. (See Table 12.) 



Table 12 



Teacher Opinions of the Quality of SIE Module Enrichment Professional Development 





Percent Agreeing 1 
(N = 57) 


There were opportunities for participants to express their views and collaborate with peers. 


95 


Adequate, time, structure, and guidance were provided for participants to discuss the SIE-supported 




modules and pedagogical strategies with each other. 


93 


The facilitators were well prepared. 


91 


Adequate time, structure, and guidance were provided for participants to reflect individually on the 




substance of this workshop. 


88 


The facilitators encouraged active participation and investigation by all participants. 


88 


The goals of this workshop were clear. 


88 


This workshop reflected careful planning and organization. 


88 


This workshop was relevant to my classroom instruction. 


86 


This workshop prepared me to use the teaching strategies promoted by SIE. 


85 


Question about the science content in the module were adequately addressed by the facilitators. 


84 


The facilitators shared tips and suggestions for successfully implementing the module. 


80 


The various components of this workshop were useful in meeting its goals. 


80 


This workshop increased my confidence in my ability to teach using the SIE module. 


80 


The facilitators explicitly discussed how, when and why to use different teaching strategies. 


78 


This workshop familiarized me with the activities and materials in the module. 


75 


The facilitators modeled effective teaching strategies. 


73 


This workshop increased my understanding of the science content in the module. 


73 


This workshop was worth the time that I invested. 


71 


I would recommend this professional development to a colleague. 


68 



Includes those teachers agreeing or strongly agreeing with each statement. 
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Although participants’ ratings of the enrichment sessions were positive, they were quite a bit 
lower than the ratings for the Initial Module Trainings. Only slightly more than two-thirds of 
respondents indicated that the workshop was worth the time invested or that they would 
recommend the workshop to colleagues (compared to nearly 90 percent for the IMTs). Most of 
the participants thought the pace of the enrichment training was appropriate, though those who 
disagreed generally indicated that the pace was too slow. (See Table 13.) 



Table 13 



Teacher Opinions of the Pace of SIE Module Enrichment Trainings 





Percent of Teachers 
(N = 57) 


Too slow 


28 


Appropriate 


70 


Too fast 


2 



Participants were also asked to reflect on which aspects of the Module Enrichment they found to 
be the most effective. The most frequent response, mentioned by about one-third of the teachers, 
was the opportunity to share their experiences and ideas for implementing the module. One-fifth 
of the respondents noted that the visual maps of the module learning goals were very helpful. 
These sentiments also came up in interviews. As three teachers described: 

I found it helpful to share ideas with my colleagues and take things they use with their 
[module title] module to use with mine in the future. It was interesting and informative 
to spend the day with [facilitator names] — both very well prepared. 

Sharing with everyone: ideas, strategies, content, assessment, materials, and classroom 
management. 

I found the visual map to be very helpful in ending me in planning my lessons so that my 
instructional tasks can lead me in the right direction to reach my learning goals. 

When asked what aspects of the session could be improved, the most common answer, given by 
one-quarter of participants responding to this open-ended question, was to spend more time 
talking about resources that could be used to supplement the modules. Other suggestions 
included increasing the amount of time for interacting with colleagues, reducing the time for 
sharing experiences and increasing the focus on the science content, though neither was given by 
more than three respondents. Example responses included: 

7 would have liked to hear more activities or supplements / could use in my classroom to 
further engage my students. 

More time to talk with teachers from other schools. 

The facilitators were dry. I would have liked more information to deepen my 
understanding of the science concepts in [Module]. 
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Teachers were also asked at the conclusion of the Module Enrichment trainings what their 
greatest concerns were about implementing the SIE-supported modules. Given the teachers’ 
level of experience and training at this point in the program, it is not surprising that about one- 
quarter of responding teachers did not identify any concerns. Of those respondents who did 
indicate some concerns, most reflected the same concerns about time and materials management 
teachers reported at the end of the Initial Module Trainings. As three wrote: 

Making sure we are able to finish each investigation before the time we need to return 
our kits. How will it effectively impact our students to their fullest potential? 

I wish we had more time in the school day. It seems like every time we turn around 
something new is added that we need to do on a daily, weekly, monthly, or yearly basis. 

Time to effectively teach the lessons appropriately. 

Observers’ Perceptions of the Quality of the Module Enrichment Trainings 

Evaluation staff observed portions of three Module Enrichment (ME) trainings. HRI utilized the 
same framework used to assess the quality of the Initial Module Trainings to examine the extent 
to which the ME professional development achieved its goals. The analysis of observation data 
from the ME Trainings focuses on the major goals of the sessions. 

Uncovering and Addressing Teachers’ Concerns about the Module 

Teachers had an opportunity at the ME sessions to voice concerns and problems they had when 
implementing the modules. They were provided time to put their questions and concerns on 
chart paper, and talk with other participants about these issues. The sessions also provided time 
for the facilitators to address the issues. For example, in one session when expressing classroom 
management concerns, a teacher asked, “What do we do if students finish early?” The 
facilitators suggested they give these students materials to sort for upcoming activities or have 
them do extra readings on the topics. In a response to a comment that the vocabulary in the 
module is too difficult, the facilitators suggested the use of a word wall, adding pictures to 
vocabulary, and making science words a part of spelling lists. 

However, there were other instances when teachers’ concerns were not addressed. For example, 
in one observed session, when issues regarding time management were raised, (e.g., the length of 
time it takes to assemble materials for an activity, or not being able to complete an activity in the 
time available for science), no helpful advice was provided. The facilitators’ comments 
included, “Yes, time is an issue,” and, “We can’t control your school’s schedule.” Although 
participants appreciated the empathy of the facilitators, it was clear that they were frustrated by 
the lack of concrete suggestions. Given the regularity with which time and materials 
management questions are raised, the program should make sure all workshop facilitators are 
prepared to address these concerns. 

There were also instances when teachers’ questions about the science content in their module 
were not adequately addressed, and in some cases, the responses they received were inaccurate 
and could lead to reinforcing common misconceptions. For example, in the Water session, when 
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discussing the expansion of water when heated, it was suggested that teachers use microwave 
popcorn to demonstrate that molecules when heated, like kernels of corn, will expand. This 
example could promote the misconception that the molecules themselves “puff up” as opposed to 
their moving farther apart. These instances reinforce the need to continue to work with 
facilitators to make sure they have a solid understanding of the science content in their module. 

Examining a Module Specific Storyline that Connects Instructional Tasks , Learning Goals, 
and Big Ideas 

After the discussion of teachers’ concerns about the module, the participants engaged in an 
activity to learn how to read a “concept map” of their module. They were given three sheets of 
paper, one with a list of instructional tasks, one with a list of learning goals, and one with a list of 
big ideas. After reviewing and coming up with a description of what was on each paper, the 
participants were given the content map for their module that linked the instructional tasks, 
learning goals, and big ideas. 

These maps provide a visual means of illustrating the science content in the module and 
providing links between the instructional tasks and the content ideas. The maps also emphasize 
that the purpose of doing the activities in the module is for students to learn science, not just for 
them to have fun. Finally, the maps show how lessons build on one another and reveal how 
skipping a lesson can get in the way of students learning a key concept. 

Although the maps have a great deal of potential, observers of the ME sessions noted that some 
of the maps contained content errors and/or omitted key ideas. For example, one of the learning 
goals on the Motion and Design map states: “Transfer of energy creates an unbalanced force.” 
Although an energy transfer can occur at the same instant a force is applied, they are both the 
result of objects interacting and one does not cause the other. The program may want to consider 
having the maps reviewed by content experts to ensure both accuracy and completeness. Ideally, 
these experts would have experience in education to make sure their suggestions are appropriate 
for teachers of elementary students. 

Establishing a Clear Understanding of What Students Are Expected to Learn in a Lesson, 

How Teachers Know What They Learned, and What Teachers Can Do If Learning Has Not 
Occurred 

After the introduction to the content map, participants engaged in a lesson-by-lesson review of 
the module in order to map specific lessons to the learning goals. The focus questions for the 
review, which participants were to respond to for each module lesson, were: 

1. What were the students to learn in this lesson/part? (this should be tied back to the 
storyline learning goals) 

2. What strategies would provide evidence that the students learned the intended 
content? 

3. What can you do if learning has not occurred? (strategies, extensions, interventions) 

The participants were divided into groups and the lessons in the module were divided up among 
the groups. The groups were assigned the task of answering the three questions listed above for 
each of their lessons. The groups worked for about one hour, recording their responses on chart 
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paper. The facilitators then led a large group sharing of each group’s findings, one lesson at a 
time, through the entire module. 

This activity was effective in helping participants understand that there are specific ideas that are 
intended to be developed in particular module lessons. The opportunity to examine each lesson, 
thinking about what activities students do and what concepts they are learning, is a valuable 
activity that many teachers do not have the time to do during their busy work schedule. Another 
strength of this activity is that it provided participants with a visual representation of how some 
concepts are addressed several times during a module while other concepts are developed in only 
one or two lessons. This observation reinforced the program’s stance that teachers should not 
skip lessons during their implementation. 

Although this portion of the workshop was well received by participants, two areas stand out as 
needing improvement. First, when participants were asked to link module lessons to the learning 
goals on the module storyline maps, they tended to link lessons to many learning goals, with 
most of the connections being tangential. For example, one group of teachers in the Weather 
Module Enrichment linked Lesson 11 “Exploring Puddles” to the learning goal “Weather may be 
measured using tools.” When the facilitator asked why, the teachers indicated that an umbrella is 
a weather tool. There was no further discussion and the group left the lesson linked to that idea. 
As a result, the activity may have been counterproductive, giving teachers the impression that all 
the activities address most of the learning goals and making it unlikely they would be able to 
help students make sense of the primary intended content. In addition, the teacher may have 
gotten the impression that skipping or modifying activities is permissible as students will learn 
all the ideas with just a few lessons. To avoid this problem, the program may want to consider 
modifying the prompt given to teachers, perhaps having them associate only those lessons that 
directly teach an idea. In addition, providing an example of an appropriate link and one of an 
inappropriate link may help participants with the activity. 

The second issue identified in the observations was that most of the teachers did not appear to 
have enough knowledge to engage meaningfully with the second and third prompt. Teachers 
misinterpreted the task and responded to the questions in a manner different than one would 
expect. For example, teachers responded to the question about assessing learning by listing 
activities from the module lesson (e.g., sort and identify cloud pictures, record cloud cover daily) 
rather than ways they could determine if students had learned the intended content. 

Teachers also appeared to misinterpret the third question, about what to do if the students had not 
learned the targeted ideas. In a Weather session, the participants shared extension activities, such 
as books to read or websites on the topic. In a Motion and Design session, teachers’ suggestions 
were more about modifying activities to make them easier for students rather than on 
interventions they could use if learning had not occurred. Teachers listed strategies like limiting 
the number of building pieces, providing a model prior to construction, coloring diagrams for 
students, adapting open-ended questions to be multiple choice, and using a line of learning. 

This portion of the workshop might benefit from the inclusion of one or two module-specific 
exemplars that model appropriate responses to these questions. An exemplar could, for a key 
lesson, outline what students should learn from the activity, point out misconceptions students 
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might have about the content, suggest a formative assessment prompt, and describe appropriate 
interventions for different student responses. To highlight the important aspects of the formative 
assessment and interventions, the workshop might also show an example that, while topically 
aligned with the targeted idea, is unlikely to elicit student ideas or provide opportunities to move 
their understanding forward. 

Discussing “ Science Misconceptions ” and Identifying Any that May Affect the Learning Goals 
of the Module 

In the observed Module Enrichment Trainings, teachers were provided an opportunity to reflect 
on the meaning of the term “science misconceptions,” how children begin to form 
misconceptions, and how teachers can address students’ science misconceptions. The discussion 
focused on science misconceptions in general and not those specific to the content of the module. 
At the end of the discussion, the teachers received a handout that included information on how 
children might form misconceptions and ways teachers can address these ideas. Participants then 
had a short discussion of student misconceptions specific to the content of the module. 

Understanding common science misconceptions is a key part of pedagogical content knowledge 
that can help teachers implement their modules more purposefully. Participating teachers likely 
benefited from the opportunity to think about the fact that students often come to school with 
incorrect ideas that get in their way of learning the correct ones. In addition, providing teachers 
with examples of misconceptions related to their particular module content both alerted them to 
the importance of paying attention to student ideas and gave them specific ideas to be on the 
lookout for. 

The participants would likely benefit from a greater focus on the misconceptions specific to the 
module content. While providing participants with a list of common misconceptions is a good 
start, a more in-depth treatment of why students might hold each of them, what students might 
say or do that would indicate they held on to these ideas, and how to specifically address each 
one would be beneficial. Of course, spending more time in the workshop on one topic would 
require a reduction in another. It will be important for the program to consider what is most 
important for teachers and what can be done well in the limited time available 

Linking Storyline Content to the PSSA 

The Module Enrichment Trainings emphasized that the content in the modules can be applied to 
help students answer PSSA questions in other content areas as well. Participants were provided 
with copies of released PSSA science items and discussed the question: “How does the teaching 
of this module assist students in the answering of these PSSA questions?” 

This activity highlighted the importance of making connections between the module activities 
and the targeted content. It also gave the message that the concepts their students are learning in 
the modules can be applied to other content/broader concepts being tested on the PSSA. Seeing 
how students can take content they learned in the module and apply it to answer test questions 
about science content not directly linked to the module may have addressed teachers’ concerns 
that teaching science with modules means covering fewer topics. 
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Impact of SIE on Teachers and Teaching 



This section of the report focuses on how the SIE program has impacted teachers, including their 
preparedness to use their SIE-provided module and their classroom teaching. Data come from 
teacher and principal questionnaires and interviews, as well as classroom observations of module 
implementation. 



Teacher Preparedness 

One of the primary goals of the SIE program is to prepare teachers to use the SIE-provided 
modules and the pedagogical approaches promoted by the program. In interviews, teachers 
indicated that the workshops were successful at preparing them to implement the modules. 
Several teachers also noted that as a result of the professional development, they adopted a new 
and more effective way to provide science instruction. In the words of two teachers: 

In a nutshell, just the whole concept of teaching science in a different way. I felt more 
prepared to walk into my classroom and actually teach what I want my students to 
learn... all the things 1 learned in the training I have used in my classroom. 

I got a better sense of how to teach science, how to better question students. 

In addition to feeling comfortable with the materials and activities in the modules, teachers need 
to know how to enact the pedagogical approaches promoted by the program. The post- 
professional development questionnaire included a series of items aimed at assessing teachers’ 
perceptions of their preparedness to use the teaching strategies promoted by SIE (e.g., the 
inquiry-based teaching strategies embedded in the SIE module, science notebooks, a learning 
cycle). To assess impacts, teachers were asked about both their current preparedness and their 
preparedness prior to the professional development session. This “retrospective pre” approach is 
useful when respondents are likely to change their perceptions of initial knowledge/preparedness 
as they learn more about the topic (i.e., they didn’t realize how much/little they knew about a 
topic until after they participated in the program). 

Responses to these items were combined into a composite variable (called “perceptions of 
pedagogical preparedness”) to reduce the unreliability associated with individual survey items. 15 
Each composite has a minimum possible score of 0 and a maximum possible score of 100. A 
score of 0 would indicate that a teacher selected the lowest response option for each item in the 
composite, whereas a score of 100 would indicate that a teacher selected the highest response 
option for each item. 



15 Definitions of this and other composites described in this report, a description of how the composites were 
created, and reliability information are included in Appendix B. 
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These longitudinal data have a nested structure, with time points nested within individual 
teachers. Statistical techniques that do not account for such nested data structures can lead to 
incorrect estimates of the relationship between independent factors and the outcome. 

Hierarchical regression modeling 16 is an appropriate technique for analyzing nested data and was 
used to examine trends in teachers’ composite scores. 

Across both workshop types, there was a significant increase in teachers’ perceptions of their 
pedagogical preparedness (See Figure 4). The mean composite score for teachers attending an 
Initial Module Training increased from 40 to 76 (a large effect 17 of 1.92 standard deviations). 
Scores for teachers attending Module Enrichment increased from 50 to 74 (a large effect of 1.27 
standard deviations). 



Teacher Perceptions of Pedagogical Preparedness 




□ Pre-PD ■ Post-PD 

* Post-workshop scores significantly different than pre -workshop scores (2-tailed paired samples t-test, 
p < 0.05). 



Figure 4 



The post-professional development questionnaire also asked teachers about their understanding 
of the student learning goals in the SIE module, the science content in the SIE module at a 
deeper level than what students are expected to learn, and ideas (either correct or incorrect) that 



lh Bryk, A.S. & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. 
Newbury Park, CA: Sage Publications. 

17 The effect size for the comparison of two adjusted means is calculated as the difference between the means 
divided by the pooled standard deviation. Effect sizes of about 0.2 are typically considered small, 0.5 medium, and 
0.8 large. Cohen, J. ( 1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence 
Erlbaum Associates. 
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students are likely to have about the content prior to instruction. These items were combined 
into a composite titled “perceptions of pedagogical content knowledge.” 

The analysis indicates that teachers’ perceptions of their pedagogical content knowledge were 
significantly greater after the SIE professional development. (See Figure 5.) Scores for teachers 
attending Initial Module Training increased from 40 to 77 points, a large effect of 1.89 standard 
deviations. Scores for teachers attending Module Enrichment increased 24 points, a large effect 
of 1.36 standard deviations. 



Teacher Perceptions of Pedagogical Content Knowledge 



2 100 -, 




Initial Module Training* Module Enrichment* 



□ Pre-PD ■ Post-PD 

* Post-workshop scores significantly different than pre-workshop scores (2-tailed paired samples t-test, 
p < 0.05). 

Figure 5 



Teachers’ Use of Instructional Strategies Promoted by SIE 

The SIE program encourages teachers to use a number of instructional strategies intended to 
enhance their implementation of the modules. In interviews, teachers were asked to comment on 
their use of several of those instructional strategies. Specifically, teachers were asked to describe 
their use of the science notebooks, focus questions, and sense-making strategies presented in the 
professional development. All of the interviewees indicated that they were using science 
notebooks in their teaching. In addition, each indicated that the use of the notebooks enhanced 
student learning. As three described: 

Our district has purchased the science notebooks. We journal, we predict, we do 
everything, we use the science journals all the time .. .the first thing we do is we write the 
lesson, we write the focus question. I have them predict and then they ’ll write their 
obserx’ations, they ’ll do scientific drawings. Sometimes they ’ll do a “before ” and they ’ll 
do an “after” drawing, a scientific drawing. 



Horizon Research, Inc. 



52 



July 2010 





We use our notebooks for every lesson, for every experiment. We always start with a 
focus question and then I have them make a prediction... and then they would put them 
aside, we would do the experiment and then we would bring them back out and look at 
the prediction and talk about what we thought was going to happen and then write in 
what did happen. 

The notebooks helped to focus them [students] on what the day’s lesson was... it really 
made them responsible, they knew they were going to be checked... They were really 
helpful. 

All of the interviewees also indicated that they use at least some of the focus questions provided 
in the professional development. As with the notebooks, teachers thought that the use of focus 
questions improved the learning experience for students. In the words of two: 

I think it really helps them focus in on the key skill or concept that we are doing for the 
day and have them really concentrate on that when we are doing the lesson as opposed to 
all the other things that could be going on. It makes them know what they need to pay 
attention to and at the end of the day they are able to say, “Okay, this is what I was 
supposed to figure out and this is what 1 did figure out. ” 

I don 7 think I would do it without using the focus questions. ..[It] sort of guides the whole 
thing, so if we didn 7 use the focus question... [they] wouldn 7 really know what we were 
doing or why we were doing it. 

Just over half of the interviewed teachers noted that they used some of the sense-making 
strategies that were presented in the trainings such as the “line of learning” or “claims and 
evidence.” These teachers indicated that these strategies help students think more deeply about 
the science ideas or helps the teacher and student know what they have learned. As two reported: 

It really forces them to think about what they did because I think often in science classes 
we would go through activities and talk about them, but we really have not taken it the 
highest level of analysis. Working with the line of learning. ..that is what it was, really 
taking the activity to the next level of analysis. 

I think the line of learning kind of sums it all up for them... It shows them what the main 
ideas or main concept is that they were supposed to get out of the experience. 

Lack of instructional time was the main reason teachers gave for not using these strategies. 



Module Implementation 

During their first year in the program, each school selects one module to implement at each 
grade level. In their second year, the school adds a second module at each grade. No new 
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modules are added in the third year, but the schools select an additional module in the fourth 
year. 



As part of the student achievement study (described more fully later in the report), teachers were 
asked how much instructional time was devoted to each module they taught, and the extent to 
which they supplemented the module with other instructional materials. As can be seen in Table 
14, teachers reported spending, on average, between 8 and 18 hours of instructional time on the 

i o 

topic covered by their module. Most of this instruction was based on the SIE-supported 
module, ranging from 67 to 96 percent of their instructional time. Taken together, these data 
indicate that teachers are basing substantial portions of their science instruction on the SIE- 
provided modules. 



Table 14 



Instruction on Topic, by Module 





Number of 
Responding 
Teachers 


Hours of 

Instructional Time 


Percent of Instructional 
Time Based on Module 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Grade 3 












Chemical Tests 


68 


10.94 


5.52 


92.34 


11.56 


Plant Growth and Development 


142 


10.21 


5.80 


84.61 


16.80 


Rocks and Minerals 


189 


11.16 


7.12 


87.87 


14.93 


Structures of Life 


17 


8.86 


7.41 


95.82 


6.87 


Grade 4 












Animal Studies 


34 


11.28 


6.09 


81.53 


21.09 


Electric Circuits 


182 


11.65 


6.82 


91.11 


13.67 


Human Body 


93 


8.10 


5.61 


86.26 


19.06 


Grade 5 












Ecosystems 


68 


14.13 


8.57 


82.99 


20.36 


Environments 


65 


10.45 


9.54 


82.78 


20.90 


Floating and Sinking 


26 


10.96 


4.92 


91.38 


15.72 


Levers and Pulleys 


51 


13.18 


9.30 


92.62 


15.82 


Motion and Design 


91 


11.73 


9.37 


85.68 


18.39 


Grade 6 












Experiments with Plants 


32 


12.15 


10.76 


67.58 


28.04 


Magnets and Motors 


17 


17.71 


10.57 


89.06 


16.75 


Mixtures and Solutions 


47 


16.72 


18.10 


88.07 


13.67 


Variables 


46 


17.25 


11.04 


87.45 


19.90 



In interviews, teachers were asked to reflect on how their implementation of their module went. 
The vast majority indicated that their implementation had gone very smoothly, with many 
specifically noting that the teacher’s manual was easy to follow and facilitated module 
implementation. Teachers’ comments included: 



18 The survey asked about only those modules related to topics on the student assessment and is not a complete list 
of modules offered by SIE. 
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It’s really amazing and the binder is terrific, with the background information and 
everything. 



I think it went really well, I think that the students enjoyed it, I enjoyed teaching it. 

Teachers also noted that having implemented a module multiple times made the process easier, 
and that their implementation improved each time. As one teacher stated: 

I feel more comfortable, it goes a lot smoother; I kind of know what to expect. The time 
to set it up is faster because I know exactly what I’m supposed to be doing instead of 
having to rely on the manuals and things like that. 

In addition, HRI administered a survey to samples of teaches at the end of the second and fourth 
quarters of the school year, asking teachers to indicate for each lesson in the module they just 
taught whether they implemented the lesson as written, implemented it with modifications, or 
skipped it. For a subset of the lessons modified or skipped, teachers were asked how/why. The 
survey also asked about teachers’ use of the reading selections and extension activities included 
in the modules. 

Overall, teachers reported implementing 50 percent of the lessons in their modules without 
modifications and an additional 29 percent with modifications. Nearly two-thirds of teachers 
reported implementing all of the lessons in their module. About 70 percent of teachers reported 
using at least one of the extension activities in their module, and roughly 90 percent indicated 
using some of the readings. 

As can be seen in Table 15, the predominant reason for not implementing a lesson, reported by 
teachers for 73 percent of skipped lessons, was not having enough instructional time. The next 
most common reasons, both given for fewer than 10 percent of skipped lessons, were that the 
lesson was too difficult conceptually or that the teacher had an alternative lesson for the same 
idea that s/he preferred. 



Table 15 



Reasons Given for Skipping Module Lessons 





Percent of Skipped 
Lessons f 


There is not enough instructional time devoted to science to implement the 




Lesson/Investigation 


73 


The Lesson/Investigation is too difficult (conceptually) for my students 


7 


I have a different Lesson/Investigation covering the same concept that I prefer using 


6 


I have classroom management concerns with the Lesson/Investigation 


5 


I don’t have the necessary equipment/supplies 


5 


It is too difficult to get the Lesson/Investigation to work as desired 


4 


The Lesson/Investigation is not engaging for my students 


2 


The Lesson/Investigation is too easy (conceptually) for my students 


2 


I am not confident in my own understanding of the science 


2 


I have safety concerns with the Lesson/Investigation 


1 


I am not confident in my ability to set up the equipment properly 


1 



’ Percentage adds to more than 100 as teachers could select multiple reasons. 
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Teacher reports of modification were more varied. About one-third of the modified lessons were 
changed to make the lesson more structured for the students. In about one-quarter of the 
modified lessons, the teacher shortened or cut-out the small group investigation portion. 
Combining a lesson with another and reducing/skipping the whole group discussion were each 
reported for about 1 in 5 of the modified lessons. (See Table 16.) 



Table 16 

Modifications Made to Module Lessons 





Percent of Modified Lessons' 


I made the Lesson/Investigation more structured to make it appropriate for my students 


31 


I shortened/cut part of the individual/small-group investigation time 


23 


I integrated pieces of this Lesson/Investigation into another one 


21 


I shortened/cut part of the whole-class discussion time 


18 


I used it as a demonstration rather than as a student activity 


15 


I made the Lesson/Investigation less structured to make it appropriate for my students 


11 


I substituted equipment/supplies 


9 



+ Percentage adds to more than 100 as teachers could select multiple modifications. 



The most common reason given for not implementing more readings and extensions was lack of 
instructional time. (See Tables 17 and 18.) Preferring to use readings/extensions from outside 
the module and the difficulty level were the next most important factors. 



Table 17 



Factors Affecting Teachers’ Use of Readings 





Not at All 


Somewhat 


Moderately 


To a Great 
Extent 


Not enough instructional time to use the reading 
selections 


10 


24 


18 


49 


Preferred using other readings 


54 


29 


12 


5 


Reading level was too difficult for my students 


67 


25 


6 


3 


The content in the readings were too difficult for my 
students 


69 


24 


5 


2 


Uncertain in my own understanding of the science 
concepts 


87 


9 


3 


0 


Reading level was too easy for my students 


92 


7 


1 


0 


The content in the readings were too easy for my students 


91 


7 


2 


0 
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Table 18 



Factors Affecting Teachers’ Use of Extension Activities 





Not at All 


Somewhat 


Moderately 


To a Great 
Extent 


Not enough instructional time to use the extensions 


4 


18 


17 


60 


Preferred using other extension activities 


49 


34 


13 


3 


The extensions are too advanced for my students 


48 


38 


11 


3 


Classroom management concerns 


64 


24 


9 


3 


Lack of the necessary equipment/supplies/materials 


66 


25 


7 


2 


Lack of connection/application to what I was teaching in 










other subject areas 


71 


22 


6 


2 


Uncertain in my own understanding of the content 


73 


18 


7 


1 


The extensions are too easy for my students 


86 


10 


3 


0 



One factor that may influence teachers’ implementation of the modules is their perceptions of 
principal support. SIE teachers were asked a series of items on the end-of-year questionnaire 
about the extent to which their principal supported their science teaching and the SIE program. 
These items were combined into a composite titled, “perceptions of principal support.” As can 
be seen in Figure 6, scores on this composite were fairly high across the four cohorts, indicating 
that teachers felt that their principals support their implementation of the SIE program. 19 
Feelings of support were higher in 2009-10 across teachers in Cohorts 1-3. 



Teacher Perceptions of Principal Support* 




□ 2007-08 □ 2008-09 ■ 2009-10 

* There was a significant different in composite scores across the time points (ANOVA, p < 
0.05). 

Figure 6 



19 This set of items was added to the end-of-year questionnaire in 2007-08; thus there is no baseline data for Cohort 
1 in this factor. 
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Teachers and principals were also asked on questionnaires about other factors that may have 
affected, positively or negatively, teachers’ use of the modules. Tables 19 and 20 show teacher 
and principal responses, respectively, to these questions over the past three years. 

Nearly all responding teachers and principals thought that the training received from SIE 
facilitated their use of the modules. In addition, most respondents indicated that the support of 
fellow teachers and the SOS facilitated use of the modules. Further, although the SIE program 
does not have any formal mechanisms for assisting teachers outside of the workshops, one- 
quarter of the teachers and more than half of principals indicated that teachers received extra 
assistance that helped them implement their module. Although there has been some variation in 
responses across the last three years, the overall pattern has been stable. 
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Table 19 

Teacher Opinions of Factors Affecting Their Use of the SIE-Provided Module, by Year 





Percent of Teachers Agreeing 




2007-08 
(N = 1,748) 


2008-09 
(N = 1,561) 


2009-10 
(N = 1,350) 


Factors Facilitating Implementation 

The training I received from SIE made it easier for me to use the 
modules. 


96 


94 


94 


It was easy to get all of the equipment and supplies necessary to use the 
modules. 






85 


Other teachers in my school provided a support system for use of the 
modules. * 2 ' 3 


86 


88 


82 


My own science background was helpful when I was teaching from the 
modules. * 1-2 


77 


82 


81 


My SOS facilitated my use of the modules. **’ 3 


73 


78 


73 


I received assistance from SIE outside of the workshop that helped me 
to use the modules. 


26 


25 


25 


Factors Inhibiting Implementation 

There was not enough instructional time for science to effectively use 
the SIE-supported modules. * 1,3 


52 


45 


53 


The amount of time required to prepare for instruction with the SIE- 
supported modules was problematic. **' 2 


55 


46 


46 


The pressures to teach mathematics and/or reading inhibited my use of 
the modules. * 2 ’ 3 


51 


51 


40 


I am not able to cover all of the science topics I am suppose to because 
of the time it took to implement an SIE module. * 2 


43 


42 


38 


I did not have the modules for long enough to use as much of them as I 
wanted to. * 1-2 ' 3 


29 


41 


37 


The time of year that I received my module was problematic. * ’ * 3 


— 


39 


33 


Too much time elapsed between the module training and module 
delivery. * 


_ 


24 


25 


The amount of time I was required to be out of the classroom was 
problematic. * 1,2 


26 


14 


14 


My lack of experience in science made it more difficult for me to teach 
from the modules. * 1- 2 


15 


10 


9 



Includes those indicating “Agree” and “Strongly Agree” on a four-point scale of 1 “Strongly Disagree” to 4 “Strongly 
Agree.” 



* The percentage of teachers agreeing with the statement differs across years (chi-square test with FDR adjustment, 
p < 0.05). 

1 2007-08 significantly different than 2008-09. 

2 2007-08 significantly different than 2009-10. 

3 2008-09 significantly different than 2009-10. 

* Item was not asked in 2007-08. 
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Table 20 



Principal Opinions of Factors Affecting 
Teacher Implementation of the SIE-Provided Modules 





Percent of Principals Agreeing 1 * 




2007-08 
(N = 93) 


2008-09 
(N = 100) 


2009-10 
(N = 112) 


Factors Facilitating Use 

Teachers provided each other with a support system for using the modules 


98 


96 


96 


The training teachers received from SIE made it easier for them to use the 
modules 


99 


99 


95 


Our Support on Site (SOS) facilitated teachers’ use of the modules 


85 


93 


92 


Teachers received assistance from SIE outside of the workshop that helped 
them to use the modules 


61 


58 


47 


Factors Inhibiting Use 

The pressures to teach mathematics and/or reading inhibited teachers’ use of 
the modules 


38 


33 


42 


The time of year that teachers received their modules was problematic® 


— 


38 


42 


The amount of time teachers were required to be out of the classroom was 
problematic 


57 


40 


40 


Teachers did not have the modules for enough time to complete all of the 
lessons 


19 


37 


36 


There was not enough science instructional time for teachers to complete the 
SIE-supported modules 


26 


38 


34 


Teachers were not able to cover all of the science topics they were supposed to 
because of the time it took to implement an SIE module 


30 


25 


38 


The amount of time required to prepare for instruction with the SIE-supported 
modules was problematic for teachers 


23 


22 


24 


Teachers' lack of experience in science made it more difficult for them to 
teach from the modules 


12 


13 


15 


Too much time elapsed between the module training and module delivery® 


— 


18 


15 


Teachers' lack of science content knowledge made it more difficult for them to 
teach from the modules 


12 


16 


14 



' Includes those indicating “Agree” and “Strongly Agree” on a four-point scale of 1 “Strongly Disagree” to 4 “Strongly 
Agree.” 



* The percentage of principals agreeing with each statement is not significantly different across years (chi-square test using 
the false discovery rate adjustment for multiple comparisons, a = 0.05) 

® Item was not asked in 2007-08. 



The data in Table 19 are further supported by responses from teachers on an open-ended 
questionnaire item. When asked what the greatest benefit of the SIE program has been, most 
teachers’ responses were the professional development and the fact that SIE provides the 
furbished modules. Teachers’ comments included the following: 

Professional development was the single most important thing that I did with the SIE 
program, with regards to teaching my children to the best of my ability! 

The greatest benefit was actually having gone through the training myself and keeping 
the notebook. I found the notebook to be a tremendous asset to my teaching. It helped 
refresh my memory and my students enjoyed seeing my examples. 

The training I received before having to teach the material to the students. Having all the 
materials needed was great. 
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Principal feedback echoed these sentiments. As two said in interviews: 

The training they go to is terrific, they love it and 1 have some seasoned teachers and they 
really enjoy it. They also come back with some new strategies. 

The PD they are required to attend makes going through the activities easy because 
they’ve done it before. So not a whole lot of confusion, they know the outcome they need 
to have. 

In terms of inhibitors, principals and teachers indicated pressure to focus on mathematics and 
reading, because those subjects count toward Adequate Yearly Progress and science does not, 
inhibited the use of the SIE-provided modules. About forty percent of both teachers and 
principals responding to questionnaires noted that pressure to teach mathematics and/or reading 
had a negative impact on use of the modules. (See Tables 19 and 20.) Interestingly, although 
still prevalent, this year fewer teachers indicated that this factor was an inhibitor than in past 
years. 

The pressure to focus on mathematics and reading is most likely responsible for the perceived 
lack of time for science instruction, which was cited as an inhibiting factor by over half of the 
teachers and one-third of the principals. In addition, 37 percent of teachers indicated that they 
did not have the modules long enough. Teachers and principals elaborated on these issues in 
interviews and in response to open-ended questionnaire items. As three reported: 

Right now, the PSSA testing schedule has really impacted us... we spent two weeks in 
testing mode which really impacted our time during that period... we lost a good two 
weeks of instruction to testing. (Teacher) 

It would be very beneficial if we could have more time allotted for use of the kit. A great 
amount of time isn’t considered due to emergencies, assemblies, testing, breaks, days off, 
etc. (Teacher) 

Time; [teachers] need more time and there is no time. Time management. They need to 
be much better time managers than they have in the past, and need to be very 
discriminating in what they do with that time. There’s no room anymore for the fluff and 
the nonsense. They really need to focus. (Principal) 

To better understand how these and other factors are influencing day-to-day instruction, HRI 
conducted 10 case studies, sampling 10 teachers from Cohorts 1-3 to participate. Teachers were 
asked to allow an observer to visit their classroom during their implementation of a lesson from a 
module, and then participate in an in-depth interview about their experiences with the module. 
The lessons observed spanned all grades K-6, and most occurred toward the end of the teachers’ 
implementation of the module. 

Instructional time for science, or the lack thereof, was a recurring theme among these teachers. 

In several cases, the lack of time was due to the scheduling of module delivery /pickup and the 
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state testing. Teachers reported that they lost several weeks of instruction due to the testing, but 
that they were not able to keep their modules for more than the allotted nine weeks. 

All of the case study teachers emphasized the impact the SIE professional development had on 
their ability to implement the module. Most indicated that they rely heavily on the science 
notebook they created at the professional development, as they were able to capture many 
suggestions for implementing the lessons. The teachers also found the information in the 
teacher’s guides to be helpful, indicating that they typically read the “background” section before 
teaching a lesson. 

Overall, the case study teachers implemented the lessons with fidelity. Several made minor 
changes to adjust the lesson for the needs of their students or to fit the lesson in the time 
available. For example, one teacher had students respond to the activity questions in a whole- 
class discussion rather than first having them write answers to the questions individually. 

Another teacher had students practice the technique they would use in the lesson prior to starting 
the investigation to ensure the activity would be successful. 

The two lessons with the biggest modifications were taught by teachers who had stronger science 
backgrounds and more experience teaching science. These teachers supplemented the materials 
in the module to help lift out the science in the activities. These lessons were among the more 
purposefully implemented ones and helped focus students on, and reinforce, the targeted science 
ideas. Appendix A includes a vignette about each teacher that illustrates the most salient factors 
affecting their module implementation. 
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Impact of SIE on Students 



Although the main emphasis of the program’s activities is on teachers and their instruction, the 
ultimate goal of the program is to improve student attitudes toward and learning of science. The 
evaluation addressed this aspect more systematically through two activities, one aimed at 
assessing the impact of the program on student attitudes toward science, the second examining 
the program’s impact on student learning. 

To assess the impact of the program on student attitudes toward science, teachers were asked a 
series of items in the end-of-year questionnaire about their students’ attitudes toward science and 
the extent to which they attributed those student attitudes to the SIE program. As can be seen in 
Table 21, the vast majority of teachers indicated that students had positive attitudes toward 
science, with nearly all respondents agreeing that students enjoyed learning about science, were 
enthusiastic about science class, were interested in science, and wanted to learn more science. In 
addition, 7 1 percent of respondents indicated that their students this year liked science more than 
previous students they had taught. In addition, teachers often attributed these positive attitudes 
to the SIE program. 



Table 21 



Teacher Report of Impacts on Students 





Percent of 


Percent Attributing 
Impact to SIE* 




Teachers 


Not 


To 


To a 




Agreeing* 


at 


some 


great 




(N = 1,348) 


all 


extent 


extent 


Students... 










Enjoyed learning about science 


99 


1 


37 


62 


Were enthusiastic about science class 


99 


2 


39 


59 


Were interested in science 


99 


2 


39 


59 


Wanted to learn more science 


97 


4 


44 


52 


Were eager to explore science on their own, outside of the classroom 


89 


11 


51 


38 


Liked science more than previous students I have taught 


71 


16 


41 


43 



Includes those teachers agreeing or strongly agreeing with each statement. 

* The percentage of teachers attributing the impact to SIE is based on the teachers agreeing with the statement. 



As part of its evaluation of the SIE program, HRI, with the assistance of the Pennsylvania 
Department of Education , designed and implemented a study to examine the impact of SIE 
professional development and teachers’ implementation of the science administered in each 
grade 3-6. Each assessment addresses content from all of the modules supported by SIE at that 
grade that focus on developing science conceptual understanding (modules that focus on 
developing science process skills were not included). 

This study sought to answer the following questions: 

1. Is there a relationship between the extent to which science instruction is based upon 
the SIE- supported modules and student achievement? 
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2. Is there a relationship between teacher participation in SIE-provided professional 
development and student achievement? 

3. Are there gender or race/ethnicity differences in student achievement, and if so, does 
the extent to which instruction is based on the SIE- supported modules and/or teacher 
participation in SIE-provided professional development reduce those differences? 



Instrumentation 

HRI developed the four assessment scales that were used in this study, with each scale 
corresponding to the science modules at a grade level provided by the SIE program. The 
modules that were addressed by the assessment at each grade level were: 

Grade 3 1. Chemical Tests (STC); 

2. Plant Growth and Development (STC); 

3. Rocks and Minerals (STC); 

4. Sound (STC); 

5. Structures of Life (FOSS); 

Grade 4 6. Animal Studies (STC); 

7. Electric Circuits (STC); 

8. Human Body (FOSS); 

9. Landforms (FOSS); 

Grade 5 10. Ecosystems (STC); 

11. Environments (FOSS); 

12. Floating and Sinking (STC); 

13. Levers and Pulleys (FOSS); 

14. Motion and Design (STC); 

Grade 6 15. Experiments with Plants (STC); 

16. Food and Nutrition (FOSS); 

17. Magnets and Motors (STC); 

18. Mixtures and Solutions (FOSS); and 

19. Variables (FOSS). 



Because of the large number of students involved in the SIE program, and the timeframe in 
which results were needed, it was also decided to use only selected-response (i.e., multiple- 
choice) items rather than including open-ended or performance items. For each grade, HRI 
developed a pool of assessment items that covered the concepts included in both the modules and 
the Pennsylvania Science and Technology Standards. All items went through a stringent, internal 
review to ensure both alignment with the targeted content and language accessibility. When 
possible, cognitive interviews were conducted with elementary students to help ensure that 
students interpreted the items as intended, and that student responses were appropriate given 
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their understanding of the content (i.e., those students who understood the content responded 
correctly, and those who did not understand the content selected an incorrect response). The 
items were then reviewed for content accuracy by Ph.D. scientists with expertise in the relevant 
topic area; any items with content issues were revised or removed from the item pool. HRI also 
sent each item pool to PDE for review and approval. These steps provide some assurance of 
content validity of the assessments. 

In addition, statistical analyses can be used to examine the validity and reliability of items. 
Factor and dimensionality analyses help determine whether a set of items form a scale (i.e., a set 
of items that measure the same ability or trait, for example knowledge of levers and pulleys). 
These analyses resulted in the dropping of a small number of items. Table 22 shows the number 
of items out of the original set for each scale that were retained based on these analyses, as well 
as the reliability for each set of items. 



Table 22 

Assessment Scale Reliabilities 





Number of Items Retained 


Reliability 


3 rd Grade 


29 of 30 


0.75 


4 th Grade 


30 of 30 


0.79 


5 th Grade 


37 of 40 


0.81 


6 th Grade 


34 of 40 


0.84 



Race/ethnicity and gender data were also collected from students. Finally, teachers provided 
information about their classes, including the amount of instruction on each of the topics on the 
assessment and the extent to which that instruction was based on an SIE-supported module. 



The Sample 

For the 2009-10 student achievement study, 38 schools from Cohort 1, 53 schools from Cohort 
2, and 15 schools from Cohort 3 were asked to participate.” Of these 106 schools, 88 returned 
pre- and post-test data in time to be included in these analyses (30 Cohort 1, 44 Cohort 2, and 14 
Cohort 3 schools). The great majority of data returned were included in the analyses; however, 
data from a number of classes were excluded due to assessment administration errors (e.g., some 
teachers did not follow the instructions for distributing answer sheets, making it impossible to 
link student pre- and post-test data for those classes). Table 23 shows the number of classes that 



20 Reliability can range from 0 to 1; typically, a reliability > 0.60 is considered acceptable for assessments used to 
evaluate a program. 

21 Although Cohort 4 schools were asked to complete the assessments, due to the late funding of the program this 
year. Cohort 4 schools were not able to complete the pre-test until January 2010 (the other cohorts completed the 
pre-test at the beginning of the school year). Because the timing of the pre-test was very different for Cohort 4, 
which could affect assessment results, it was decided to exclude Cohort 4 data from these analyses. 
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were expected to be part of the study for each assessment scale, as well as the number of classes 
included in the final analyses. 



Table 23 

Number of Classes in the Study, by Cohort 





Cohort 1 


Cohort 2 


Cohort 3 


Expected 


Included in 

Analyses 


Expected 


Included in 

Analyses 


Expected 


Included in 

Analyses 


3 rd Grade 


129 


74 


192 


136 


53 


47 


4 th Grade 


145 


99 


200 


142 


58 


49 


5 th Grade 


138 


104 


185 


146 


57 


39 


6 th Grade 


101 


84 


111 


86 


37 


24 



Table 24 provides demographic information for the students included in these analyses. Overall, 
the classrooms contained about the same number of females and males. Most students classified 
themselves as White; nearly all students indicated that English was their primary language. 



Table 24 

Student Demographic Data 





Percent of Students 




3 rd Grade 


4 th Grade 


5 th Grade 


6 th Grade 




(N = 4,125) 


(N = 5,136) 


(N = 5,041) 


(N = 3,402) 


Gender 










Female 


48 


50 


50 


50 


Male 


52 


50 


50 


50 


Race/Ethnicity' 










American Indian/ Alaskan Native 


1 


1 


1 


1 


Asian 


1 


1 


1 


1 


B lack/ African- American 


9 


10 


11 


11 


Hispanic/Latino 


1 


2 


3 


3 


Native Hawaiian/Other Pacific Islander 


0 


0 


0 


0 


White 


89 


88 


87 


87 


English is Primary Language 


99 


99 


99 


99 



The total percentage may add to more than 100 as students could select more than one category. 



Analysis and Results 

Item response theory (IRT) was used to score the assessments. IRT takes into account the 
relative difficulty of each item such that more difficult items contribute more to students’ scores. 
In addition, using IRT removes the error associated with day-to-day fluctuations in student 
performance, and is a better estimate of student knowledge than a raw score such as number or 
percent correct. Scores were calculated on a scale of 0 to 100. Table 25 shows the scores for 
each assessment scale. 
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Table 25 

Student Assessment Scores 





Minimum 


Maximum 


Mean 


Standard 

Deviation 


3 rd Grade 










Pre-Test 


0 


71 


30.12 


12.93 


Post-Test 


0 


100 


44.27 


16.95 


4"' Grade 










Pre-Test 


0 


86 


35.67 


14.63 


Post-Test 


0 


100 


48.58 


16.06 


5 th Grade 










Pre-Test 


0 


84 


36.70 


14.94 


Post-Test 


0 


100 


46.88 


17.58 


6"' Grade 










Pre-Test 


0 


100 


43.48 


16.31 


Post-Test 


0 


99 


51.94 


17.98 



The student assessment data have a nested structure, with test administration (pre-test and post- 
test) nested within students nested within classes. Consequently, HLM was used to analyze the 
data. Results for each model follow, organized by research question. Regression coefficients 
and standard errors are presented in Appendix C. 

> Is there a relationship between the extent to which science instruction is based upon the 
SIE- supported modules and student achievement? 

To answer this question, HRI used data from the surveys administered to teachers at the middle 
and end of the school year.' These surveys asked teachers which SIE-supported modules they 
implemented, how many science lessons they taught on the assessed topics, and what proportion 
of instructional time was based on the modules. These data were used to create three variables 
for the analyses that summarize instruction on the topics covered by the assessment: the number 
of topics for which the teacher provided instruction; the amount of instructional time using SIE- 
supported modules; and the amount of instructional time using non-SIE materials. As can be 
seen in Table 26, the amount of science instruction on the tested topics was relatively low, 
ranging from an average of about 16 hours over the course of the year in 4 th grade to roughly 30 
hours in 6 th grade. There was a fair amount of variation in instructional time though, as 
evidenced by the relatively large standard deviations. In addition, the high maximum values 
indicate that some teachers spent a great deal of time teaching the assessed topics. 



22 Because not all teachers completed these surveys, the number of classes included in this analysis is smaller than 
the total number of classes returning pre- and post-test data. 
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Table 26 

Science Instruction in SIE Classes 





Minimum 


Maximum 


Mean 


Standard 

Deviation 


3 rd Grade (N = 213) 










Number of Assessed Topics Taught 


0 


5 


2.53 


1.18 


Instructional Hours Using SIE-Supported Modules 


0 


77 


18.23 


13.70 


Instructional Hours Using Non-SIE Materials 


0 


27 


4.46 


6.21 


4 th Grade (N = 213) 










Number of Assessed Topics Taught 


0 


4 


2.80 


1.00 


Instructional Hours Using SIE-Supported Modules 


0 


65 


15.74 


12.28 


Instructional Hours Using Non-SIE Materials 


0 


55 


7.57 


10.32 


5 lh Grade (N = 197) 










Number of Assessed Topics Taught 


0 


5 


3.20 


1.29 


Instructional Hours Using SIE-Supported Modules 


0 


105 


26.06 


21.82 


Instructional Hours Using Non-SIE Materials 


0 


43 


6.07 


8.29 


6 lh Grade (N = 122) 










Number of Assessed Topics Taught 


0 


5 


2.47 


1.09 


Instructional Hours Using SIE-Supported Modules 


0 


104 


30.17 


22.89 


Instructional Hours Using Non-SIE Materials 


0 


39 


8.09 


10.48 



The regression models examined changes from pre- to post-test scores, and the extent to which 
the three variables characterizing teachers’ instruction related to those changes. The models also 
examined whether achievement gaps existed by gender and race/ethnicity, and if there was 
adequate class-level variation, the models investigated whether the instructional variables were 
related to changes in any existing gaps. Because the number of students classifying themselves 
as any race/ethnic group other than White was small, the data were collapsed into two categories: 
White/Asian vs. non-Asian minority. 23 

Because classes are not randomly assigned to different levels of instruction on the assessed 
topics, it is possible that any differences in post-test scores may be due to initial differences 
among classes. To examine this possibility, the analyses examined the initial equivalence of 
classes relative to the instructional variables included in the study. In grades 3 and 5, there were 
no significant differences overall on pre-test scores among classes with different amounts of 
instruction based on SIE-supported modules. There were some differences in pre-test scores in 
grades 4 and 6. In both grades, students in classes covering a greater number of assessed topics 
had lower pre-test scores. In grade 6, students in classes that experienced more of their science 
instruction from SIE-supported modules had slightly higher pre-test scores. In addition, non- 
Asian minorities scored 6 to 8 points lower on the pre-test than their White/Asian classmates at 
each grade level. 

Post-test scores were significantly higher than pre-test scores at all four grade levels, ranging 
from an 8-point increase in grade 6 to a nearly 14-point increase in grade 3. More importantly, 
the amount of growth was, at each grade, significantly related to the amount of instructional time 
based on the SIE-supported modules. Table 27 shows how a typical student would be expected 



; i Asian students typically outperform all other groups of students, and are often grouped with White students in 
these types of analyses. 
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to score in classes receiving different amounts of instruction based on the SIE-supported 
modules. These results show a positive relationship between the amount of instructional time 
based on SIE-supported modules and student achievement, though the effect is relatively small 
(effect sizes less than 0.20). 



Table 27 

Expected Post- Test Scores, 

by Amount of Instructional Time Based on SIE-Supported Modules 





1 Standard Deviation 
Below the Mean Amount 
of Instructional Time 


Mean Amount of 
Instructional Time 


1 Standard Deviation 
Above the Mean Amount 
of Instruction Time 


3 rd Grade 


40.87 


44.12 


46.22 l 


4 th Grade 


47.71 


48.02 


48.22 


5 th Grade 


43.91 


46.22 


47.59 


6 th Grade 


45.37 


51.51 


55.45 



The non- Asian minority vs. White/Asian achievement gap did not change significantly from pre- 
to post-test in grades 3 and 6. The gap widened slightly in grades 4 and 5. The amount of 
instructional time based on SIE-supported modules was not related to this change. 

> Is there a relationship between teacher participation in SIE-provided professional 
development and student achievement? 

HRI also examined whether the extent of teacher participation in SIE-provided professional 
development was related to student achievement. For these analyses, HRI used program records 
of teacher attendance at SIE Initial Module Trainings and Module Enrichment sessions to 
classify teachers by the number of trainings they attended related to the topics assessed. Because 
there was limited variation in the professional development attendance at each grade level, HRI 
combined data across the four grade levels to increase the statistical power (i.e., the probability 
of detecting a difference if one truly exists) of the analysis. As can be seen in Table 28, the 
majority of grades 3-6 teachers in this study have attended an Initial Module Training, and 
almost half have attended two or more; about one-quarter of the teachers have attended a Module 
Enrichment session. 



Table 28 



Teacher Attendance of SIE Professional Development 





Percent of Teachers 
(N = 747) 


Initial Module Trainings 




0 


7 


1 


46 


2 


37 


3 or more 


11 


Module Enrichment 




0 


77 


1 


23 
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The analysis found that students of teachers who had attended two or more Initial Module 
Trainings had significantly greater growth from the pre- to post-test than students of teachers 
who had attended fewer of these sessions. Effect sizes were relatively small (i.e., under 0.20 
standard deviations). There were no differences between classes of teachers who had attended a 
Module Enrichment session and those who had not. Figures 7-10 show test scores of typical 
students at each grade level for teachers with different levels of attendance at SIE-provided 
professional development. 



Grade 3 Test Scores, by SIE PD Attendance 




□ Pre □ Post 



Figure 7 



24 Specifically, students of teachers who attended three IMTs had greater growth than students of teachers who 
attended zero IMTs. In addition, students of teachers who attended two or three IMTs had greater growth than 
students of teachers who attended one IMT. 
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Grade 4 Test Scores, by SIE PD Attendance 




□ Pre DPost 



Figure 8 



Grade 5 Test Scores, by SIE PD Attendance 




□ Pre □ Post 



Figure 9 
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Grade 6 Test Scores, by SIE PD Attendance 




□ Pre DPost 



Figure 10 
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Summary and Recommendations 



In its fourth year, the SIE program continues to make progress in bringing inquiry-based science 
instruction to students across Pennsylvania. The program has added a fourth cohort of schools 
and provided a Strategic Planning Institute for those schools. In terms of teacher training, SIE 
has provided three days of professional development to Cohort 4 teachers, and has continued to 
offer professional development to teachers in Cohorts 1-3. In addition, the program has 
provided leadership training for a new cadre of teacher leaders and assisted schools with 
conducting their Showcase of Student Learning. 

By providing these services, SIE has played a key role in helping participating districts make 
progress toward updating their science programs. Evaluation data indicate that participating 
schools have increased the amount of instructional time devoted to science. In addition, SIE has 
helped schools shift their science instruction from a predominantly textbook-/worksheet-based 
program to a more hands-on, inquiry-based one. However, although SIE (and the introduction of 
the science PSSA) has increased emphasis on science, administrators in participating schools 
also indicated that science continues to take a backseat to reading and mathematics as those are 
the subjects that count toward Adequate Yearly Progress. 

Overall, the professional development provided by SIE has been well received by both teachers 
and principals. Participating teachers reported that the Initial Module Trainings had given them 
the knowledge and skills they need to implement a science module for the first time. The 
trainings have also generated a great deal of enthusiasm among teachers for teaching science, a 
subject many have shied away from in the past. Teachers who have attended Module 
Enrichment Training indicated that these sessions have been helpful in improving their 
implementation of the module. Participants in both types of professional development have had 
an increase in their perceptions of pedagogical preparedness and pedagogical content knowledge. 

Classroom observations of teachers implementing the modules provided additional insight into 
the impacts of the program. The enthusiasm generated in the professional development has 
carried through into the classroom. In all of the observed lessons, students were excitedly 
conducting the hands-on science activities, collecting and recording data and discussing what 
they observed. The classroom observations also revealed that, while the process of improving 
science instruction has begun, teachers need continued support to complete the transformation. 
Although farther along the continuum to purposeful implementation than non-teacher leaders, 
teacher leaders also appear to need additional support to reach this goal. Both teacher leaders 
and other teachers would likely benefit from additional professional development on the science 
content in the module and how to best use the activities to help students engage with and make 
sense of the targeted science ideas. 

In terms of student outcomes, there are multiple pieces of evidence that the SIE program is 
having positive impacts. Teachers reported a high level of student interest in and enjoyment of 
science, and the vast majority of teachers attributed these positive attitudes to the SIE program. 

In addition, analyses of student achievement data found a positive relationship between the 
extent to which teachers base their science instruction on the SIE-supported modules and student 
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learning in science. The analyses also found that, on average, the more SIE-provided 
professional development teachers participate in, the greater the change in their students’ scores 
from the pre- to post-tests. 

To assist the program in achieving its goal of continuous improvement, HRI offers the following 
rec ommendations : 

> Continue developing all SIE professional development providers’ understanding of, and 
skill at implementing, the “learner hat/teacher hat” approach. 

Overall, the Initial Module Trainings have been successful at preparing teachers to implement the 
modules at a mechanical level. These trainings have provided teachers with ample experiences 
to engage with the set-up and implementation of the module activities. However, classroom 
observations have found that teachers struggle with providing appropriate sense- making 
opportunities for their students. To help teachers implement the modules more purposefully, the 
program has adopted the learner hat/teacher hat model for the Initial Module Trainings. Still, the 
effectiveness of its implementation varied greatly. It may be that, as facilitators gain more 
experience with this approach, their implementation of it will improve. This process would be 
helped by providing facilitators with opportunities for feedback and reflection. 

Another reason for the variation in quality of implementation may be that facilitators feel 
obligated to have participants experience every lesson in the module as learners, leaving little 
time for discussion of teaching issues beyond logistical ones. One solution might be to use this 
approach only for selected lessons: those that cover science ideas that are particularly 
challenging and/or important, and those that are the most difficult to implement from a logistics 
standpoint. For other lessons, facilitators would go faster, perhaps simply describing the lesson 
briefly or asking participants to review it on their own (e.g., in the evening after the first day). If 
the program decides to take this approach, it will be important that a detailed plan for the Initial 
Module Training for each module be developed and that all facilitators be expected to follow the 
plan for their module. 

> Continue to refine the Module Enrichment Trainings so that they focus on the skills and 
knowledge that will be most beneficial to teachers of the SIE-supported modules. 

The Module Enrichment Trainings helped teachers feel better prepared to teach the module and 
use the instructional strategies promoted by the SIE program. However, there is not enough time 
in the sessions to address each of its goals in sufficient depth. While each goal has value, it may 
be better to postpone treatment of some of them in order to address others more effectively. The 
program should choose those goals that it believes will have the biggest impact on teachers’ 
ability to implement the module purposefully. Data from the classroom observations suggest that 
developing teachers understanding of the science content in the module and what each activity 
adds to the content storyline would be good candidates for additional emphasis. 

> Increase opportunities for teacher leaders to deepen their understanding of the content 
in their module, as well as its content storyline. 

Teacher leaders play a very important role in SIE, allowing the program to provide professional 
development to a greater number of teachers. Because of the multiplying effect of teacher 
leaders (each teacher leader will train many teachers), the potential for impacting the entire 
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program in a positive or negative manner is great. In order for the program to reach its goal of 
preparing teachers to provide high quality science instruction, it must ensure that the people 
leading the professional development are able to lead workshops effectively. Many of the 
teacher leaders, new and veteran, have a tenuous understanding of the science content in their 
module. Finding ways to further support the teacher leaders in this area should be a high priority 
for the program. 

> Consider ways to help teachers find the time needed to implement the modules well. 

Lack of instructional time for science is a recurring theme in much of the evaluation data. The 
SIE program may want to consider ways that it can help teacher make the most of the time they 
have. When creating the schedule for module delivery and return, the program should be sure to 
take into account major events that would prevent teachers from using the modules, including 
holidays and PSSA testing. 

The program may also want to consider creating an implementation guide for each module that 
illustrates how the module could be used most effectively with different amounts of instructional 
time. Doing so would help ensure that teachers are selecting the most important activities to 
complete when they are forced to pick and choose due to time constraints. Identifying which 
lessons are most important for helping students master the state’s science standards would be a 
good approach for developing these guides. 
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Appendix A 

Case Study Vignettes’ 



1 Pseudonyms are used to protect teacher identities. 
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Vignette 1: Using Science to Teach Other Subjects 



Mr. Taylor is a veteran primary school teacher who has been teaching for 32 years, 20 of which have been as a Kindergarten 
teacher. He has addressed science in his daily instruction for approximately ten years. His school is in its second year of 
participation in SIE, and Mr. Taylor uses two SIE-supported science modules in his instruction. The observation took place 
in the spring of 2010 during his implementation of the FOSS Wood and Paper module; he had implemented this module 
once before, in the previous year. 

Science instruction is a high priority for Mr. Taylor. Although it is not mandated by the district to do so, he teaches science 
three times a week for approximately 45 minutes. As with most primary teachers, Mr. Taylor teaches in a self-contained 
classroom and has some flexibility in his daily schedule, which allows him the opportunity to adjust the amount of time 
devoted to science instruction. 

With the exception of a few science education courses, Mr. Taylor has had no formal science coursework or instruction 
during his adult education. Even though he has limited coursework in science, Mr. Taylor indicated that he feels confident 
with the science ideas addressed in the Wood and Paper module. When asked if there were any ideas or lessons that he 
found difficult to address, Mr. Taylor indicated that there was nothing in the module that was conceptually difficult, either 
for himself or his students. However, he did note that because he has such a large classroom and no teaching aides, often 
the logistics of implementing the module lessons are challenging. 

Having attended only the Initial Module Training for the Wood and Paper module, Mr. Taylor indicated that the training 
was extremely helpful in familiarizing him with how to implement the module. He specifically noted that walking through 
the lessons as a learner helped him better understand how to address each lesson with his own students. He further noted 
that as he prepared to implement the module lessons in his classroom, he relied heavily on both the notebook he prepared at 
the training and the teacher’s manual. These two materials are the only things he uses to prepare for teaching with the 
module, indicating that the teacher’s manual is particularly helpful. 

When asked to reflect on the quality of the instructional materials provided by SIE, Mr. Taylor indicated that he really likes 
the modules. He appreciates that all of the materials are provided and notes that this availability makes doing hands-on 
science much easier. In the past, the majority of his science instruction was text-based and any materials he had for 
activities were those that he brought in himself. He also noted that he likes the readings provided in the module, stating that 
he has addressed all of the Science Stories in the Wood and Paper module. Remarking that Kindergarten students are doing 
more and more reading now, Mr. Taylor indicated that he tries to bring in additional readings that relate to the topics in the 
modules. 

In addition to the readings, Mr. Taylor makes an effort to incorporate some of the extension activities into his classroom. In 
fact, the observed lesson was an art extension activity. In Investigation 4.2 students had constructed paper mache bowls. 
The observed lesson involved students painting and decorating their bowls. 

With the exception of additional readings, Mr. Taylor indicated that he teaches the module lessons as they are written. 
However, he did indicate that he skips around in the module. In his classroom, several students had special needs and/or 
behavioral problems. The varying abilities in his classroom contributed to his shuffling the modules lessons, picking those 
lessons that he thought would be easier for students to engage with to do first. When discussing the varying abilities of his 
students, Mr. Taylor made the point that the modules seem to level the playing field for students with special needs. For 
example, he noted that one student struggles to stay abreast with his contemporaries in reading and writing and gets very 
frustrated. However, in science, this student excels and is clearly more engaged. 

Overall, Mr. Taylor is implementing the Wood and Paper lessons as they are written, but out of sequence. He is making 
cross-curricular connections to the module in other subjects, such as reading and art. The greatest influences on his 
preparations for implementing the module are the materials he received from SIE, specifically his notebook and teacher’s 
manual. Next year, Mr. Taylor indicated that he would like to try to implement the module in sequence and as written and 
noted that he will feel more confident to do so having implemented the module two times at that point. 
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Vignette 2: Making Changes to Increase Student Interest 



Ms. Martin has taught music for much of the past 25 years. For the last two years she has been at her current school as a 
learning support teacher for students in Kindergarten, 1 st , and 2 nd grade. She is in her second year of using two SIE- 
supported science modules, which she typically teaches in collaboration with a 1 st grade teacher in the classroom next door. 
The first year using the modules, Ms. Martin provided assistance as the 1 st grade teacher led the lessons. This year, the two 
teachers took turns leading the lessons. 

Ms. Martin has the flexibility to decide when to teach science and for how long. She indicated that she is able to use any 
instructional materials she wants, but that she exclusively uses the SIE-supported modules to teach science. When she has a 
module, Ms. Martin teaches science every day for approximately 45-60 minutes. When she does not have a module she 
instead teaches social studies. 

Before coming to her current school, Ms. Martin did not teach science. She has not taken any college science courses, and 
the only science professional development she has attended is the Initial Module Training for the Weather module. Ms. 
Martin commented that she enjoyed the training and that it was very helpful for preparing her to teach the module. She also 
indicated that the module itself is very easy to use. She noted that the module is accessible to her students because it is 
hands-on and allows them to learn through their own observations. Ms. Martin has especially embraced the Initial Module 
Training emphasis on having students make observations using all five senses. 

One factor that impacts Ms. Martin's implementation of the module is the amount of time she is allowed to keep it. Ms. 
Martin reported that, even though the module was in the school for nine weeks, because of the PSSA she had only a few 
weeks in which to implement the module. She indicated that she had to rush to get through all of the investigations in this 
short period of time. To make the most out of the time she had with the module, Ms. Martin collaborated with the three 1 st 
grade teachers in the building to plan how they would implement the module. They decided to combine some lessons, 
shorten others, and cut some out entirely. She indicated that their decisions were based on what they thought were the most 
important things for the students to learn. For example, they combined several lessons that focused on thermometers into 
one, choosing to have students measure temperature only in degrees Fahrenheit (excluding temperature measurement in 
degrees Celsius). The limited time also prohibited her from using any of the extension activities in the module. 

A second factor that impacts Ms. Martin's implementation of the module is the diverse learning needs of her students. Ms. 
Martin is a learning support teacher and many of her students have unique learning styles as well as behavioral issues. 

Thus, she modifies the instruction based upon her perceptions of student ability and interest. For instance, she often spends 
extra time on the reading and literacy components of the science lessons because she believes this supplemental instruction 
will help them in their lives. She also skips many of the provided module readings in favor of her own reading selections 
that she thinks will be more interesting to her students. 

Ms. Martin was observed teaching Investigation 12: Testing Rainy Day Fabrics from the Weather module. She generally 
followed the experimental procedures as written with a limited number of modifications. For example, Ms. Martin 
extended the lesson introduction in order to incorporate some literacy strategies. She wrote the list of materials on a flip 
chart and asked students to help her read the words aloud. She also helped students sound out each word and then held up 
the corresponding object once they read the word correctly. The Initial Module Training’s emphasis on making 
observations with all five senses was evident in this lesson; she prompted students to describe the various fabrics used in 
the investigation in terms of how they felt, what they looked like, and what they smelled like. 

Ms. Martin's focus on maintaining student interest was very evident toward the end of the lesson, which included more 
substantial modifications of the lesson. One change was dining the class discussion of their results. Rather than using the 
prompts provided in the module that focused on which types of fabric keep people dry in the rain, she questioned students 
about which types of fabric they would like to wear in the rain. As a result, the students were unable to explain which 
fabric was the most waterproof (one of the goals of the lesson). Additionally, rather than using the module reading that also 
focused on types of material used to keep humans and animals dry in the rain, she chose to read from a story about a dry 
place where everyone longed for rain and felt happy when it arrived. She thought the brightly-colored pictures in the story 
would hold student attention better than the module reading. 
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Vignette 3: First Timer Learning the Ropes 



Ms. Butler has been a teacher for four years, including two years at her current school. The majority of her teaching career 
has been in a 6 th grade classroom, but four months into this school year she voluntarily switched from 6 th to 2 nd grade to fill 
a teaching vacancy. She admitted that it was difficult to make this transition, but she made the change because she wanted 
to help fill a need at her school. 

This year was her first time teaching with an SIE-supported module. Ms. Butler has very little formal training in science 
content other than one science teaching class in her Master’s program and the Initial Module Training for the Changes 
module. She never taught science before this year, though she indicated that she feels confident to teach the content 
because of the way the module is structured. She said that the Initial Module Training was extremely helpful in preparing 
her to teach because everything was modeled and she was able to experience the investigations step-by-step. Ms. Butler 
stated that teaching the module is, “Kind of like a no-brainer if you remember what happened” in the training. During the 
Initial Module Training, Ms. Butler also created a science notebook that she uses to help her prepare to teach the lessons. 

Ms. Butler has complete control over the science instruction and science curriculum in her classroom. She typically teaches 
science three or four times each week for up to 45 minutes per lesson. Additionally, she plans all of her own instruction. 
Before Ms. Butler received the Changes module, she taught science lessons that she pieced together by consulting assorted 
books and websites. She described the struggle to find science topics and activities that she was confident and competent 
teaching. She is very happy with the Changes module because it has eliminated much of her anxiety about teaching 
science. She indicated that the module is teacher-friendly and is not a burden for her to prepare because everything is 
provided. 

Ms. Butler strives to implement the module as it is written. She teaches the module lessons in order and does not make 
modifications to individual lessons. Although she occasionally selects an additional reading, she always has her students 
use the module readings. To ensure that she is following the module dining a lesson, Ms. Butler prepares to teach each 
lesson by reading the teacher background information and then writing the steps of the investigation on an index card. She 
uses this index card as a reminder of the lesson procedure while she is teaching. 

Ms. Butler was observed while teaching Investigation 10: Separating Mixtures of Color from the Changes module. 
Adherence to the module was evident during classroom observation as she methodically led students through the 
investigation. For instance, Ms. Butler had students count to four aloud as they made ink dots on filter paper and then blow 
on the dots to help them dry. This procedure was performed exactly as written in the teacher’s manual. Ms. Butler was also 
observed holding an index card throughout the lesson and she referred to it for guidance several times as students 
transitioned from one step to the next. 

In accordance with the Initial Module Training she attended, Ms. Butler typically instructs her students to write predictions 
and observations in their notebooks before, during, and after each science lesson. However, the students did not use their 
notebooks during the observed lesson. Several students pointed out that the notebooks were missing and Ms. Butler 
explained that she chose not use them due to the water and permanent markers being used in the investigation. She instead 
asked students to write on worksheets that would later be stapled into their notebooks. This modification seemed 
appropriate for the given situation. 

Although Ms. Butler followed the experimental procedures as written, she altered the introduction of the observed lesson 
from what was provided in the module. Ms. Butler began the hands-on investigation without explaining that ink could be 
separated into its hidden colors. While students were able to identify specific colors they saw after water was dropped onto 
the ink dots, it was unclear whether or not they understood that ink was a mixture of these different colors. The omission of 
this one idea may have made it difficult for students to connect the changes they observed in the activity to the larger idea 
of colors being mixtures. Ms. Butler is typical of a teacher new to teaching science with modules. She was very concerned 
with implementing the mechanics of the module activity, but would benefit from guidance on how to focus students on the 
relevant aspects of the lesson, in this case pointing out to students that the ink is an example of a mixture of liquids. 
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Vignette 4: Combining Lessons to Complete the Module in the Allotted Time 



Mr. Smith has been teaching 3 ld grade for his entire four-year career at the same school. He teaches a self-contained class 
of 25 students, several of whom are pulled out to receive Title 1 reading instruction on Tuesdays and Thursdays. Because 
he does not want students to miss science, Mr. Smith chooses to teach science on Mondays, Wednesdays, and Fridays; 
lessons are typically 45 minutes each. 

This year is Mr. Smith’s second teaching the STC Plant Growth and Development module. Mr. Smith believes he is 
adequately prepared to teach the module despite having only one general science course in college. He indicated that the 
Initial Module Training and the teacher’s manual provided the content knowledge needed to teach the module. He 
highlighted the influence of the Initial Module Training on his use of the module, in particular the emphasis at the 
workshop placed on the importance of going through every lesson in the module sequentially. In addition, he noted that the 
nature of the module content (i.e., following the growth cycle of the plants) does not permit him to rearrange or skip the 
lessons. However, due to his decision to teach science only when his entire class was present, he does not have the module 
long enough to do every lesson as written before having to return the module. As a result, he has chosen to combine three 
similar lessons about pollination rather than not get to the end of the module. Otherwise, Mr. Smith tries to implement each 
lesson as laid out in the module. 

During the observation, Mr. Smith taught Lesson 15 of the Plant Growth and Development module, which focuses on 
interpreting information from graphs. Students were given two growth charts. One was a bar graph of the Wisconsin Fast 
Plant they had been working with that illustrated the height of the plants over time; the other was a bar graph showing 
height versus time of a fictional girl on a farm. Following a set of questions on a module worksheet, students examined the 
graphs, noting important aspects such as axis titles and units of measurements. They also examined trends in the data to 
draw conclusions. Mr. Smith followed the lesson nearly exactly as written in the module, making only minor modifications 
to allow the class to finish the lesson in the time allotted. For example, rather than having students work on answering the 
questions individually, he had them working on this task as a whole class. These modifications did not appear to hinder the 
students’ opportunity to reach the objective of the lesson, as Mr. Smith made a point to call on most of the students in the 
class during the discussion and allowed the students to formulate responses themselves. The whole-class discussion likely 
contributed to students' knowledge on how to interpret graphs. 

When asked what he hoped his students would know after completing the module, Mr. Smith explained that he hoped his 
students would understand the entire life cycle of the plant from a seed growing into a plant and ultimately producing more 
seeds. He also hoped students would understand the different factors involved in that process, such as the role bees play in 
pollination. He thinks the majority of his students will arrive at these understandings because, in his opinion, the module 
does a good job of communicating the objectives in a way students can understand while challenging students in critical 
thinking and problem solving skills. 
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Vignette 5: Beefing Up the Content in the Module 



Mr. Anderson has been teaching science for 15 years, and has been at his current school teaching 4 th grade for the last three 
years. He teaches science to two different groups of students, with each group engaging in the same lesson each day. This 
is his second year teaching the FOSS Human Body module. 

Mr. Anderson reported having a strong science content background, and places a high priority on continuing to develop his 
disciplinary content knowledge. In his undergraduate studies, Mr. Anderson took every opportunity to study science, taking 
biology, chemistry, earth and space science as his elective courses. He also attended both the Initial Module Training and 
the Module Enrichment Training to become more familiar with the SIE-supported modules he teaches as well as the science 
content in the modules. Mr. Anderson indicated he is very comfortable with the content of the modules he teaches. 

When asked about influences on his teaching of the modules, Mr. Anderson stated that the biggest issue is not being given 
enough time to complete the modules. Typically, Mr. Anderson's school has the module for 13 weeks, but this year his 
school received the spring module a month late, and his implementation was further delayed due to state testing. Because 
of these delays, Mr. Anderson chose to make significant modifications to the implementation of the Human Body module. 

Mr. Anderson relied on his own science content background and the state science standards when deciding how to modify 
his use of the module in order to meet time limitations. He compared the state’s standards to the module lesson content and 
chose to implement those lessons he believed meet the standards. Mr. Anderson provided the example that he determined 
that his students needed to cover the fourth lesson in the Human Body module, which covers the function of joints and the 
opposable thumb in particular, to meet a state standard. In order to accomplish this, he condensed Lessons 1-3 into a single 
lesson (deciding those lessons were not as vital) and used it as an introduction to Lesson 4. 

Mr. Anderson also supplements the module with resources that he believes benefit his students. In particular, he places a 
high importance on technology in the classroom and uses a SmartBoard in most classes. He also brings in visual aids. For 
example, to supplement a lesson on joints, Mr. Anderson brought an actual cow knee joint into his classroom. 

Mr. Anderson's modifications were evident in the observed lesson. The main objective of the lesson was for students to 
gain an understanding of a bone’s function based on its shape and size (i.e., form relates to function). According to the 
manual, students are to take a package of model rodent bones and sort them according to an illustrated worksheet. Next, 
students are to reconstruct and display a skeleton using glue to attach the model bones. Finally, students are instructed to 
make entries to their word banks and answer a series of questions such as, “How are bones of rodents like those of humans? 
How are they different?” and “Can you tell the function of a bone by its shape?” 

The teacher made several changes to this design. First, students were questioned about bone shapes and names during the 
sorting activity, which is beyond the scope of the lesson. Next, the students assembled the bones as intended, and sketched 
the bone structures (they did not glue the bones together, most likely to save time). Mr. Anderson supplemented the lesson 
by showing the skeleton of the animal on the SmartBoard and discussing how the bones allowed specialized functionality in 
the animal. These modifications, in particular the more in-depth discussion about the functionality of bones, most likely 
increased students’ opportunity to learn the content. 
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Vignette 6: Simplifying Lessons and Adding Content for Students 



Ms. Barber has been teaching 4 th grade math and science for four of her five years teaching, all of which have been at her 
current school. This is her second year teaching the FOSS Motion and Design module, though she has taught the content 
included in the module for all five years of her career. Ms. Barber attended the Initial Module Training, and she indicated 
she feels “somewhat comfortable” teaching the module content. She has also gained content knowledge by reading 
textbooks and the internet. She teaches science to two different groups of students four days a week, with each class 
approximately 45 minutes in length. 

Ms. Barber noted that she was instructed during the Initial Module Training to teach all the module lessons in order, going 
through each lesson step-by-step. She indicated that after teaching the module the first year, she found it necessary to 
modify and/or supplement certain aspects of the lessons to better meet the needs of her students. Therefore, lesson 
modifications are a regular part of her implementation of the module. For example, in Lesson 8 students work with a 
rubber-band powered vehicle and are instructed to narrow the frame of the vehicle in order to increase friction. Ms. Barber 
explained that it is difficult for students to clearly observe this change and now leaves this portion of the lesson out. Instead 
she uses her own experiment about friction, having students slide objects across different surfaces, including ice, tile, and 
carpet. She also includes a lesson on Newton’s three laws of motion prior to starting the module to make sure students 
understand the science. 

Another area in which Ms. Barber supplements the module is assessment. Her school requires the assignment of grades; 
therefore, she has created a rubric for assessing students on the lessons. Time is also a factor affecting Ms. Barber’s use of 
the module. She indicated several lessons that she finds difficult to teach in a single science period. As a result, she makes 
modifications such as splitting lessons over two days, and having students answer questions in groups rather than 
individually to save time. Despite the additional lessons and the modifications to the module, Ms. Barber indicated she is 
able to finish the module in the allotted time. 

Ms. Barber did not make any major modifications on the day of observation, during which she taught the final lesson in the 
Motion and Design module. In this lesson, which is not typical of the other lessons in the module, student teams present 
their solutions to vehicle design challenges they had been previously assigned (e.g., design a vehicle that will move 100 cm 
in the least amount of time). The rest of the class then evaluates each team’s solution and asks questions. Ms. Barber 
followed the lesson plan fairly closely, though she did place more emphasis than called for in the module on having 
students in the classroom ask questions. Typical questions students asked included, “Why did you name your vehicle the 
way you did?” “What was the hardest part?” and “Did you have fun?” 
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Vignette 7: The Class’ Ecosystem Disturbed by State Testing 



Ms. Gilbert is a veteran teacher, with 25 years of classroom experience. Fourteen years of her career have been devoted to 
teaching the 5 th grade. Ms. Gilbert has an extensive background in science education, in particular environmental science, 
and has a passion to foster in her students an appreciation for the world around them. Ms. Gilbert holds two degrees, one in 
elementary education and the other in environmental education. She has taken upwards of 10 college courses that relate to 
the content addressed in the Ecosystems module. Ms. Gilbert has always made science a priority in her instruction by 
ensuring that it is part of her daily schedule. She stated that she has addressed science every day for the duration of her 
time as a 5 th grade teacher. 

When preparing for the lessons, Ms. Gilbert indicated that she relies on her teacher’s manual, the student activity sheets, the 
notebook she prepared during the Initial Module Training for this module, as well as the textbook used in her classroom 
when they do not have an SIE-supported module. Ms. Gilbert noted that she uses all of these materials to identify the ideas 
addressed pulling in as much detail as she can. When there are connections that she can make in their regular textbook, she 
does so, often in the form of readings. Ms. Gilbert noted that she accesses the teacher background information in the 
module to brush up on the content and vocabulary. She finds the student activity sheets to be particularly helpful when 
preparing for lessons because they focus on what the student needs to do. 

In discussing her implementation of the Ecosystems module, Ms. Gilbert indicated that she teaches the lessons in order and 
for the most part as they are written. She noted that it would not make sense to skip around in the module because the 
lessons build on one another. She stated that she sometimes makes minor adjustments to the lessons when she wants to 
highlight a particular idea. 

Ms. Gilbert noted that time for science is always a concern. At her school, there is a block of time set aside for science each 
day, but it is up to individual teachers to decide when to use that block of time. Ms. Gilbert’s block of time for science is 
bisected by lunch, which she said often results in her having to break up the lessons and make adjustments as to how much 
she can get through in one day. In addition, the modules were delivered during the PSSA testing period this year, resulting 
in a loss of three weeks of instructional time with the module. Ms. Gilbert noted that she struggles to get through all of the 
lessons in the time available; she has not had the opportunity to use any of the module extensions or strategies introduced at 
the trainings, such as Power Conclusions, because of her compressed timetable. She did note however, that she makes the 
point to address all of the readings in the module and coordinates as much as she can with her reading program. This class 
will get through as much of the module as they can before the end of the year, but Ms. Gilbert noted they will not finish 
given that they lost so much time due to state testing. 

The observed lesson was Lesson 8: Upsetting the Stability, in the Ecosystems module. Prior to the observation, students 
had constructed eco-columns by connecting two soda bottles. In the upper soda bottle, they constructed a terrarium and in 
the lower bottle they had constructed an aquarium. In the observed lesson, students made some observations of their eco- 
columns and discussed what a disturbed ecosystem is and what factors might cause an ecosystem to become disturbed. 

Ms. Gilbert’s passion for science was apparent in her instruction. Throughout the observed lesson Ms. Gilbert made an 
effort to honor students’ observations and inspire a sense of wonder about what was happening in their eco-columns by 
prompting careful and thoughtful observation. Moreover, she encouraged the students to take ownership of their learning 
by suggesting that they write everything they observed down and then do some research on their own about what they 
observed if they had any questions. 

There were minor adjustments made in the observed lesson that allowed the teacher to expand on particular ideas. One 
adjustment was that the teacher had students list and share both natural and human-made disturbances to an ecosystem. 

(The lesson as designed has the class list human-made disturbances only.) Another adjustment the teacher made was to not 
tell the students which liquid they were testing was water and which was vinegar and hope they figure it out by measuring 
of the pH of each. Other than these few minor adjustments, the teacher implemented the lesson as written in the module. 
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Vignette 8: Adjusting Implementation for Low- Ability Students 



Ms. Davis has been teaching 5 th grade for her entire eight-year tenure as a teacher; in fact, she has been in the same 
classroom throughout her career. As a member of a Cohort 3 school, Ms. Davis has been involved in the SIE program for 
two years and implemented two modules this year. Ms. Davis was observed as she addressed a lesson from the FOSS Ideas 
and Inventions module, a module she implemented once before, during her first year in the SIE program. 

Although finding time to fit science instruction into the daily schedule is often a struggle for teachers, Ms. Davis makes an 
effort to address science every day. As a result of participating in the SIE program, the administration at Ms. Davis’s school 
saw the need to increase the amount of time devoted to science instruction. As of this year, science and social studies are 
taught on a rotating six-day block schedule; science is taught every other day for one hour. On those days when it is not on 
the block schedule, science is taught for 30 minutes. 

In addition to the increase in time devoted to science in her school's daily schedule, another major influence on Ms. Davis’s 
science instruction has been the adoption of the SIE-supported modules. Ms. Davis indicated that the teacher manuals are 
very easy to follow. She had attended the Initial Module Training for both of the modules she teaches and commented that 
the training for the Ideas and Inventions module in particular was very thorough and helpful. 

When asked how she prepared for implementing lessons from the module, Ms. Davis indicated that she relied heavily on 
her teacher’s manual, science notebook, and colleagues. Her science notebook was created at the Initial Module Training 
for this module and includes the tips and strategies she gleaned from the training along with her own thoughts. In addition, 
the teachers at this school have shared grade-level planning time that Ms. Davis indicated is very valuable to her. During 
this time, she and her colleagues sit down with their materials and go through each lesson, discussing how they are going to 
implement them and what alterations they may need to make for particular groups of students. Ms. Davis also noted that 
she accesses the teacher background information in the teacher’s manual to prepare her to answer students’ questions about 
the content. 

Of the many factors that affected Ms. Davis’s implementation of the Ideas and Inventions module, the most salient factor 
was her students’ reading ability. At her school, students are grouped based on their reading ability. Ms. Davis teaches the 
group with relatively low reading ability, requiring substantial extra support and structure compared to the other students in 
the school. As a result, Ms. Davis has found it necessary to adjust her instruction with the SIE-supported modules. 

Although she implements the investigations as written and in sequence, she found that she has had to slow down the pace 
and split investigations across several class periods. Ms. Davis also noted that she has had to provide extra support in terms 
of materials management and notebook entries. For example, she often goes through the lesson directions with her students 
and has them use highlighters to emphasize the most important portions. Another thing she noted doing was having 
students practice several of the techniques used in the module investigations before actually doing the investigation. These 
extra structures were clearly evident in the observed investigation. Although Ms. Davis followed the manual and 
implemented the investigation as written, she adjusted how the materials were distributed and walked students through the 
activity providing extensive support with the directions. 

When asked about the readings and extension activities provided in the module Ms. Davis noted that she really liked the 
readings and included them all. She also noted that when it was appropriate she would bring in supplemental readings that 
related to the content of the module. For example, she had her students read a book about inventions by children at the 
beginning of the Ideas and Inventions module. She also made an effort to incorporate at least one extension activity per 
investigation. Because she did not feel she could give up class time to address them, and to keep the investigations as 
straightforward as possible, Ms. Davis selected those extensions that students could do on their own and assigned them as 
homework. Other than additional readings and the extension activities she selected in the module, Ms. Davis did not 
supplement the module with any other instructional materials. In fact, due to her slowed pace and because the module was 
delivered a bit late, she doesn’t think she will be able to get through the last set of investigations this year. 
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Vignette 9: Fidelity to Module/Flexibility to Meet Student Abilities 



Mr. Jones has been teaching for 29 of his 32-year career at his current school. For the last five years, he has taught 5 th grade 
science and mathematics to both classes of 5 th graders in his school. (The other 5 th grade teacher is responsible for 
reading/language arts and social studies). His school is in its third year of participation in SIE, and Mr. Jones teaches two 
SIE-supported science modules. The observation for this case study took place in the spring while he was implementing a 
lesson from the FOSS Variables module. This year was his second time teaching this module. 

Mr. Jones had taken some science courses in college, and felt confident in his understanding of the science topics he 
teaches. He had attended both the Initial Module Training and the Module Enrichment sessions for the Variables module, 
and indicated that both were extremely helpful in understanding and teaching the module. He also reported that both 
professional development sessions emphasized the importance of implementing all of the lessons in the module, in the 
order they are written, and that he had taken this message to heart. 

When asked his opinion of the module, Mr. Jones indicated that it is very well organized, and the background material 
included is very thorough and easy to find. In addition, he noted that the individual lessons are well structured, and that the 
module comes with all of the supplies he needs, making it easy for him to implement each lesson. However, he thought the 
module lacked adequate resources for assessing student learning. He indicated that parents in his district want to know why 
students receive the grades they do, and wanted to see student scores from assignments and tests. Mr. Jones noted that 
FOSS sells additional assessment materials, which he thinks are of high quality and would like to use, but that these 
materials are not included with the modules he receives from SIE. 

Other than adding assessments, Mr. Jones indicated that he does not feel the need to supplement the module. He noted that 
his implementation of the module is aided by the flexibility he has in his day-to-day schedule. He normally has 30 minutes 
a day, four days a week, for science. However, he and the social studies teacher are able to trade time — on some days he 
will take more than his allotted time for science, and will give the time back on other days — allowing him to complete 
lessons from the module that take more than 30 minutes to implement. On Fridays, the school has “integrated language 
arts,” and the teachers on each grade-level team share the time to further student literacy skills. By utilizing some of this 
time, he is able to implement a couple of the extension activities or reading selections included with the module. 

One factor that has a large impact on Mr. Jones’ implementation of the module is the number of weeks he is allowed to keep 
the module. Although having a module for nine weeks is sufficient during the fall semester, he indicated that it is not 
sufficient time to implement an entire module in the spring when the administration of the state assessments takes away 
large chunks of instructional time. In this situation, he noted that he still implements the lessons in order, getting as far in 
the module as he can. For the Variables module, this meant getting through less than three-quarters of the module. 

The ability level of a class also affects his implementation of the module. Of the two groups of 5 th grade students Mr. Jones 
works with, he described one group as being more advanced than the other, noting that they take more initiative during 
experiments and need less guidance. Although Mr. Jones indicated that he follows the module exactly as it is presented in 
the teacher’s manual, without skipping any lessons, he thinks it is okay to make modifications to individual lessons to fit 
the needs of a particular group of students. 

Modifications to meet the needs of a particular class were evident on the day of the observation, during which Mr. Jones 
was teaching Lesson 2 of Investigation 3: Plane Sense to his advanced class. The lesson continued an experiment in which 
students had built a rubber-band powered airplane and were investigating the plane’s flight along a piece of fishing line. 

The lesson plan called for small groups to first determine how many times they had to wind the rubber band to cause the 
plane to travel the entire length of the string. Next, the students were to regroup as a class and discuss their findings, 
reviewing the different variables that may have affected the flight of the planes. Students were then supposed to go back to 
their small groups to determine how many rubber-band winds it would take to fly the plane half-way across the string. 
Because he was teaching his more advanced class, Mr. Jones decided to combine these three steps into one, having students 
determine the number of winds in takes to move the plane both distances without the intermediate discussion. For this 
class, the approach taken by Mr. Jones appeared to be effective. Though the step-by-step procedures from the manual were 
not followed verbatim, the teacher’s modifications to the lesson seemed appropriate as students engaged with the all of the 
important aspects of the lesson. 
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Vignette 10: Teacher Modifications Help Students Focus on Content 

Mr. Garner is an experienced teacher of 35 years, 13 of which has been spent teaching 6 th grade. Mr. Garner holds a 
bachelor's degree in elementary education, which included taking several science education courses, but not any science 
courses. Even with his lack of formal education in science, Mr. Garner indicated that he has always been very interested in 
and passionate about science. He has been involved in several science education initiatives in his school over his career and 
has taught science every year in his daily instruction since becoming an elementary educator. Currently, Mr. Garner teaches 
science every day for approximately 50 minutes, despite the pressure from his principal to spend more time on reading and 
mathematics. 

Although he has implemented the FOSS Mixtures and Solutions module only twice since his school joined SIE, Mr. Garner 
indicated that he is particularly comfortable with the content addressed given that he has taught the topic many times prior 
to receiving the SIE-supported module. Mr. Garner attended both the Initial Module Training and the Module Enrichment 
Training for the Mixtures and Solutions module, which he stated has further increased his confidence with the content in the 
module. He noted that when preparing for a lesson in this module, he typically looks at all of the materials in his teacher’s 
manual and the notebooks he created during each SIE training on this module. He noted that the notebooks are particularly 
helpful in providing implementation tips and strategies, and that the teacher background section in the module is also 
particularly helpful for refreshing his memory on the content. 

Mr. Garner’s philosophy as to how students learn best influences his implementation of the module. He suggested that 
students often have to see things for themselves to believe it, and that as a teacher he can not just tell the students something 
and expect them to understand it. He explained, “This process takes time;” and, although he typically follows the manual 
very closely, he often has to break lessons up across multiple class periods. Typically, in one class period Mr. Garner will 
provide the class a focus question, a list of vocabulary, and have students make predictions. In the next class period, he 
continues the lesson with the activity. Noting that he really likes the open-ended nature of the questions in the module, Mr. 
Garner also breaks up lessons to free up time for more discussions about the class’s findings. He particularly likes the 
questions in the module because they place the onus for coming to a conclusion on the students, pushing them to think more 
deeply about their findings and the key ideas in the module. 

Mr. Garner did note that his students struggled with some concepts in the module, which also influences his implementation 
of the module. He indicated that the ideas most difficult for his students were those related to what a solution is, that 
solutions can not be separated by filtration, and what a saturated solution is. Mr. Garner noted that those lessons dealing 
with saturation and solutions were not only conceptually difficult for students, but were also logistically challenging for the 
instructor and were very material-intensive. He plans time to reinforce those ideas once he knows how students respond to 
the lessons. 

Given the way Mr. Garner spreads out lessons across several class periods, he expects to finish the third set of 
investigations in the Mixtures and Solutions module but not get to the fourth and final set that address chemical reactions. 
He indicated that both times he has implemented this module he has not gotten to the fourth set of investigations. 

Other than introducing the vocabulary at the beginning of the lesson rather than at the end, as the module suggests, Mr. 
Garner stated that he implements the module lessons in sequence and as they are written. He does not add much to the 
module, especially given his already limited amount of time for science. In an effort to make cross-curricular connections 
to reading and mathematics, he has students do in-depth analyses of the Science Stories that come with the module. Mr. 
Garner also noted that he has been unable to incorporate many of the module extensions into his implementation and 
addresses only those that he has materials readily available for and that are easy to do. Typically, he assigns the extensions 
as extra-credit assignments. 

The observed lesson was Investigation 3.1 from the Mixtures and Solutions module. Concentration: Soft Drink Recipes. 

The purpose of this lesson is for students to explore the concept of concentration (i.e., amount of solute dissolved in a given 
volume of solvent). 

The observed lesson followed the structure as outlined in the module, but Mr. Garner added a few elements that helped 
students focus on the targeted content. For example, they used a balance to compare the weights of two pitchers prior to 
and after different amounts of a soft-drink powder had been added to each. Moreover, the teacher involved the students 
more directly in the investigation than outlined in the module by calling on pairs of students to come to the front to make 
the solutions. Throughout the lesson, the teacher kept the focus on the targeted ideas by asking questions like, “Is this a 
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solution?” and, “Can I filter it?” At the conclusion of the investigation, the teacher had prepared a worksheet and overhead 
he had prepared that that had images of solutions with dots representing the solute dissolved, underneath was a continuum 
showing that the solution was either more or less concentrated and more or less dilute moving in one direction. Students 
were instructed to pick whether the solutions were getting more or less concentrated and more or less diluted based on the 
direction of the arrow. This activity was a way for students to visualize the inverse relationship between concentrated and 
diluted. These modifications appealed effective at helping students focus on the targeted idea. 
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Composite Definitions 
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To facilitate the reporting of large amounts of survey data, and because individual questionnaire 
items are potentially unreliable, groups of survey questions that measure similar ideas can be 
combined into “composites.” Each composite represents an important construct related to 
science teaching or professional development. Cronbach’s Coefficient Alpha is a measure of the 
reliability of a composite (i.e., the extent to which the items appear to be measuring the same 
construct). A Cronbach’s Alpha of 0.6 is considered acceptable, 0.7 fair, 0.8 good, and 0.9 
excellent. 

Each composite is calculated by summing the responses to the items associated with that 
composite and then dividing by the total points possible. In order for the composites to be on a 
100-point scale, the lowest response option on each scale was set to 0. As a result, someone who 
marks the lowest point on every item in a composite receives a score of 0, and someone who 
marks the highest point on every item receives a score of 100. It also assures that 50 is the true 
mid-point. The denominator for each composite is determined by computing the maximum 
possible sum of responses for a series of items and dividing by 100; e.g., a nine-item composite 
where each item is on a scale of 0-4 would have a denominator of 0.36. 



Table B-l 



Composite: Teacher Perceptions of Pedagogical Preparedness 



Preparedness to: 


Post-PD 

(Prior) 


Post-PD 

(Now) 


Use the inquiry-based teaching strategies embedded in the SIE module 


Q8a-p 


Q8a-n 


Use science notebooks to support student learning of the content in the SIE module 


Q8b-p 


Q8b-n 


Use the FERA (Focus, Explore, Reflect, Apply) and 5 E (Engagement, Exploration, 
Explanation, Elaboration, Evaluation) Learning Cycles to teach using the SIE 
module 


Q8c-p 


Q8c-n 


Manage the logistics of the SIE module 


Q8d-p 


Q8d-n 


Handle classroom management issues with the SIE module 


Q8e-p 


Q8e-n 


Use questioning strategies to elicit student thinking about the science concepts in the 
module. 


Q8f-p 


Q8f-n 


Examine student work to assess student thinking about the science concept in the 
module. 


Q8g-p 


Q8g-n 


Teach the science concepts addressed in the module. 


Q8h-p 


Q8h-n 








Number of Items in Composite 


8 


8 


Reliability (Cronbach’s Coefficient Alpha) 


0.93 


0.94 
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Table B-2 



Composite: Teacher Perceptions of Pedagogical Content Knowledge 



Understanding of: 


Post-PD 

(Prior) 


Post-PD 

(Now) 


Student learning goals (big ideas) in the SIE module 


Q7a-p 


Q7a-n 


Science content in the SIE module at the level that the students are expected to learn 
it 


Q7b-p 


Q7b-n 


Science content in the SIE module at a deeper level than what students are expected 
to leam 


Q7c-p 


Q7c-n 


Ideas (either correct or incorrect) that students are likely to have about the content in 
the SIE module before instruction 


Q7d-p 


Q7d-n 


How the activities in the module connect conceptually with one another. 


Q7e-p 


Q7e-n 


How the activities in the module contribute to understanding the big ideas of the 
module. 


Q7f-p 


Q7f-n 


Real-world connections to the science content in the module. 


Q7g-P 


Q7g-n 








Number of Items in Composite 


7 


7 


Reliability (Cronbach’s Coefficient Alpha) 


0.96 


0.93 



Table B-3 



Composite: Teacher Perceptions of Principal Support 



My Principal... 


End-of Year 
Questionnaire 


Makes attending the SIE professional development a priority 


Q14a 


Provides opportunities for teachers participating in SIE to meet and share ideas 


Q14b 


Is not very knowledgeable about the SIE program 


Q14c 


Is enthusiastic about the SIE program 


Q14d 


Makes science teaching a priority 


Q14e 


Is supportive of teachers participating in SIE 


Q14f 


Encourages implementation of the SIE modules 


Q14g 


Encourages the use of innovative science instructional strategies 


Q14h 


Accepts the noise associated with the activity based SIE modules 


Q14i 


Encourages teachers to integrate science and literacy 


Q14J 






Number of Items in Composite 


10 


Reliability (Cronbach’s Coefficient Alpha) 


0.92 
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Appendix C 



Student Assessment 

Regression Coefficients and Standard Errors 
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Table C-l 

HLM Student Assessment Analysis Results: Module Use 







Regression Coefficients (and standard errors) 






Grade 3 


Grade 4 


Grade 5 


Grade 6 


Intercept (pre-test score) 


29.79 


(0.32) 


34.86 


(0.36) 


35.86 


(0.47) 


42.61 


(0.72) 


Number of Topics Taught 


-0.23 


(0.35) 


-0.88* 


(0.44) 


0.11 


(0.42) 


-1.71* 


(0.76) 


Square Root of Hours of Instruction Using 
SIE Modules 


-0.08 


(0.21) 


-0.39 


(0.22) 


0.20 


(0.23) 


1.50* 


(0.43) 


Cube Root of Hours of Instruction Using 
Other Materials 


0.13 


(0.44) 


1.89* 


(0.48) 


-0.03 


(0.56) 


-2.18* 


(0.96) 


Female 


-0.06 


(0.46) 


-0.90 


(0.46) 


-0.96 


(0.49) 


0.61 


(0.67) 


Non- Asian Minority 


-5.83* 


(0.84) 


-6.23* 


(0.79) 


-7.90* 


(0.87) 


-7.45* 


(1.16) 


Growth from Pre to Post 


13.82* 


(0.44) 


13.11* 


(0.33) 


10.01* 


(0.40) 


7.95* 


(0.42) 


Number of Topics Covered 


0.40 


(0.48) 


0.04 


(0.40) 


0.14 


(0.36) 


1.15* 


(0.44) 


Square Root of Hours of Instruction Using 
SIE Modules 


1.60* 


(0.28) 


0.54* 


(0.20) 


0.56* 


(0.20) 


0.70* 


(0.26) 


Cube Root of Hours of Instruction Using 
Other Materials 


1.14 


(0.61) 


0.27 


(0.44) 


-0.48 


(0.48) 


-0.98 


(0.56) 


Female 


0.08 


(0.45) 


0.62 


(0.40) 


-0.14 


(0.48) 


0.12 


(0.55) 


Number of Topics Covered 


— 


— 


— 


— 


0.53 


(0.40) 


— 


— 


Square Root of Hours of Instruction Using 
SIE Modules 










-0.18 


(0.22) 






Cube Root of Hours of Instruction Using 
Other Materials 










-0.44 


(0.53) 






Non- Asian Minority 


-1.62 


(0.89) 


-2.91* 


(0.70) 


-1.75* 


(0.75) 


-0.94 


(0.89) 



* p < 0.05. 

Note: All variables were grand-mean centered except the Growth from Pre to Post variable which was uncentered. 
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Table C-2 

HLM Student Assessment Analysis Results: 
Teacher Participation in SIE Professional Development 





Regression Coefficients (and standard errors) 


Intercept (pre-test score) 


-0.37 


(0.01) 


Grade Level 1 






Grade 3 


-0.05 


(0.04) 


Grade 5 


0.11* 


(0.04) 


Grade 6 


0.17* 


(0.04) 


Initial Module Training 2 






1 Initial Module Training 


0.02 


(0.06) 


2 Initial Module Trainings 


-0.11* 


(0.06) 


3+ Initial Module Trainings 


-0.08 


(0.07) 


Module Enrichment Training 


0.03 


(0.03) 


Female 


-0.02 


(0.01) 


Non-Asian Minority 


-0.36* 


(0.03) 


Growth from Pre to Post 


0.71* 


(0.01) 


Grade Level 1 






Grade 3 


0.08* 


(0.03) 


Grade 5 


-0.21* 


(0.03) 


Grade 6 


-0.30* 


(0.04) 


Initial Module Training 2 






1 Initial Module Training 


-0.01 


(0.05) 


2 Initial Module Trainings 


0.08 


(0.05) 


3+ Initial Module Trainings 


0.16* 


(0.06) 


Module Enrichment Training 


0.03 


(0.03) 


Female 


0.01 


(0.01) 


Grade Level 1 






Grade 3 


0.01 


(0.04) 


Grade 5 


-0.02 


(0.04) 


Grade 6 


-0.03 


(0.04) 


Initial Module Training 2 






1 Initial Module Training 


-0.03 


(0.06) 


2 Initial Module Trainings 


0.02 


(0.05) 


3+ Initial Module Trainings 


-0.12 


(0.06) 


Module Enrichment Training 


-0.03 


(0.03) 


Non-Asian Minority 


-0.11* 


(0.02) 



* p < 0.05. 

Note: All variables were grand-mean centered except the Growth from Pre to Post. 

1 Versus Grade 4 students 

2 Versus teachers attending no Initial Module Trainings 
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